Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove unnecessary device queries #620

Merged
merged 8 commits into from
Mar 28, 2025
Merged

Remove unnecessary device queries #620

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
79b0f6e
Rewrote ParseDevice, ParsePrecison and OpenVINOExecutionProvider code…
nproshun Mar 18, 2025
14eeee8
Addressed issues and added a minor change in ParseDeviceType.
nproshun Mar 24, 2025
9b9cabd
Merge branch 'ovep-develop' into remove_device_queries
nproshun Mar 24, 2025
a66ee24
Fixed bug in OpenVINOExecutionProvider constructor.
nproshun Mar 24, 2025
534b19f
Added default precision for composite and custom devices. Modified check
nproshun Mar 26, 2025
10794a1
Merge branch 'ovep-develop' into remove_device_queries
nproshun Mar 26, 2025
3307b88
Fixed multiple GPU logic to allow choosing just GPU without index in a
nproshun Mar 26, 2025
87f74d5
Merge branch 'ovep-develop' into remove_device_queries
nproshun Mar 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 37 additions & 23 deletions onnxruntime/core/providers/openvino/openvino_execution_provider.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,36 +62,50 @@ OpenVINOExecutionProvider::OpenVINOExecutionProvider(const ProviderInfo& info, s
// to check if target device is available
// using OVCore capability GetAvailableDevices to fetch list of devices plugged in
if (info.cache_dir.empty()) {
bool device_found = false;
std::vector<std::string> available_devices = OVCore::Get()->GetAvailableDevices();
bool all_devices_found = false;
// Checking for device_type configuration
if (info.device_type != "") {
if (info.device_type.find("HETERO") != std::string::npos ||
info.device_type.find("MULTI") != std::string::npos ||
info.device_type.find("AUTO") != std::string::npos) {
device_found = true;
std::vector<std::string> devices_to_check;
if (info.device_type.find("HETERO:") == 0 ||
info.device_type.find("MULTI:") == 0 ||
info.device_type.find("BATCH:") == 0 ||
info.device_type.find("AUTO:") == 0) {
auto delimit = info.device_type.find(":");
const auto& devices = info.device_type.substr(delimit + 1);
devices_to_check = split(devices, ',');
} else {
for (const std::string& device : available_devices) {
if (device.rfind(info.device_type, 0) == 0) {
if (info.device_type.find("GPU") != std::string::npos && (info.precision == "FP32" ||
info.precision == "FP16" ||
info.precision == "ACCURACY")) {
device_found = true;
break;
}
if (info.device_type == "CPU" && (info.precision == "FP32")) {
device_found = true;
break;
}
if (info.device_type.find("NPU") != std::string::npos) {
device_found = true;
break;
}
devices_to_check.push_back(info.device_type);
}

// Re-initialize before loop
all_devices_found = true;
for (const auto& device : devices_to_check) {
bool device_found = false;
std::string device_prefix = device;
int device_idx = 0;
// Get the index and remove the index from device_prefix
if (auto delimit = device_prefix.find("."); delimit != std::string::npos) {
try {
device_idx = std::stoi(device_prefix.substr(delimit + 1));
} catch (std::exception& ex) {
ORT_THROW("[ERROR] [OpenVINO] Wrong index in specified device - " + device + " :", ex.what());
}
device_prefix = device_prefix.substr(0, delimit);
}
std::vector<std::string> available_devices = OVCore::Get()->GetAvailableDevices(device_prefix);
// If idx is 0, maybe index is not set (e.g. GPU)
// Then the device is found if we have at least one device of the type
if (device_idx == 0 && available_devices.size() >= 1) {
device_found = true;
} else {
// Find full device (e.g GPU.1) in the list
if (std::find(std::begin(available_devices), std::end(available_devices), device) != std::end(available_devices))
device_found = true;
}
all_devices_found = all_devices_found && device_found;
}
}
if (!device_found) {
if (!all_devices_found) {
ORT_THROW("[ERROR] [OpenVINO] Specified device - " + info.device_type + " is not available");
}
}
Expand Down
136 changes: 85 additions & 51 deletions onnxruntime/core/providers/openvino/openvino_parser_utils.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,74 +10,108 @@
const std::string& option_name) {
using DeviceName = std::string;
using DefaultValue = std::string;
using ValidValues = std::list<std::string>;
using foo = std::pair<DefaultValue, ValidValues>;
using ParserHelper = std::map<DeviceName, foo>;

using ValidValues = std::vector<std::string>;

Check warning on line 13 in onnxruntime/core/providers/openvino/openvino_parser_utils.cc

View workflow job for this annotation

GitHub Actions / Optional Lint C++ 

[cpplint] reported by reviewdog 🐶
 Add #include <vector> for vector<>  [build/include_what_you_use] [4]

Raw Output:
onnxruntime/core/providers/openvino/openvino_parser_utils.cc:13:  Add #include <vector> for vector<>  [build/include_what_you_use] [4]
using DefaultAndValidPair = std::pair<DefaultValue, ValidValues>;

Check warning on line 14 in onnxruntime/core/providers/openvino/openvino_parser_utils.cc

View workflow job for this annotation

GitHub Actions / Optional Lint C++ 

[cpplint] reported by reviewdog 🐶
 Add #include <utility> for pair<>  [build/include_what_you_use] [4]

Raw Output:
onnxruntime/core/providers/openvino/openvino_parser_utils.cc:14:  Add #include <utility> for pair<>  [build/include_what_you_use] [4]
using ParserHelper = std::unordered_map<DeviceName, DefaultAndValidPair>;

Check warning on line 15 in onnxruntime/core/providers/openvino/openvino_parser_utils.cc

View workflow job for this annotation

GitHub Actions / Optional Lint C++ 

[cpplint] reported by reviewdog 🐶
 Add #include <unordered_map> for unordered_map<>  [build/include_what_you_use] [4]

Raw Output:
onnxruntime/core/providers/openvino/openvino_parser_utils.cc:15:  Add #include <unordered_map> for unordered_map<>  [build/include_what_you_use] [4]
// {Device prefix, {Default precision, {Supported precisions}}}
ParserHelper helper = {
{"GPU", {"FP16", {"FP16", "FP32", "ACCURACY"}}},
{"NPU", {"FP16", {"FP16", "ACCURACY"}}},
{"CPU", {"FP32", {"FP32", "ACCURACY"}}},
};

std::set<std::string> deprecated_device_types = {
"CPU_FP32", "GPU_FP32", "GPU.0_FP32", "GPU.1_FP32", "GPU_FP16",
"GPU.0_FP16", "GPU.1_FP16"};

bool is_composite = device_type.find(':') != std::string::npos; // FOR devices AUTO:,HETERO:,MULTI:

if (provider_options.contains(option_name)) {
const auto& precision = provider_options.at(option_name);

if (is_composite) {
std::set<std::string> allowed_precisions = {"FP16", "FP32", "ACCURACY"};
if (allowed_precisions.contains(precision)) {
// If we have multiple device configuration, request precision from user and check it
if ((device_type.find("HETERO:") == 0) ||
(device_type.find("MULTI:") == 0) ||
(device_type.find("BATCH:") == 0) ||
(device_type.find("AUTO:") == 0)) {
if (!provider_options.contains(option_name)) {
LOGS_DEFAULT(INFO) << "[OpenVINO] Precision is not set. Using default OpenVINO precision for " + device_type + ". \n";
return "";
} else {
std::unordered_set<std::string> supported_precisions = {"FP16", "FP32", "ACCURACY"};

Check warning on line 32 in onnxruntime/core/providers/openvino/openvino_parser_utils.cc

View workflow job for this annotation

GitHub Actions / Optional Lint C++ 

[cpplint] reported by reviewdog 🐶
 Add #include <unordered_set> for unordered_set<>  [build/include_what_you_use] [4]

Raw Output:
onnxruntime/core/providers/openvino/openvino_parser_utils.cc:32:  Add #include <unordered_set> for unordered_set<>  [build/include_what_you_use] [4]
std::string precision = provider_options.at(option_name);
if (supported_precisions.contains(precision)) {
return precision;
} else {
ORT_THROW("[ERROR] [OpenVINO] Unsupported inference precision is selected. ",
precision, ".\n");
ORT_THROW("[ERROR] [OpenVINO] Unsupported precision for the ", device_type, " device. Device supports only FP16, FP32, ACCURACY.\n");
}
}
}

// Deprecated device specification (CPU_FP32, GPU.0_FP32, etc.)
if (auto delimit = device_type.find("_"); delimit != std::string::npos) {
if (provider_options.contains(option_name)) {
ORT_THROW("[ERROR] [OpenVINO] Precision is specified twice, please remove the _precision suffix from device name and only set the precision separately.\n");
}
LOGS_DEFAULT(WARNING) << "[OpenVINO] Selected 'device_type' " + device_type + " is deprecated. \n"
<< "Update the 'device_type' to specified types 'CPU', 'GPU', 'GPU.0', "
<< "'GPU.1', 'NPU' or from"
<< " HETERO/MULTI/AUTO/BATCH options and set 'precision' separately. \n";
std::string precision = device_type.substr(delimit + 1);
// Device type is updated in-place
device_type = device_type.substr(0, delimit);
// We have to remove the index (.0, .1, etc.) to use device as key for helper
std::string device_prefix = device_type;
if (auto dot_delimit = device_prefix.find("."); dot_delimit != std::string::npos) {
device_prefix = device_prefix.substr(0, dot_delimit);
}

if (!helper.contains(device_prefix)) {
ORT_THROW("[ERROR] [OpenVINO] Selected 'device_type' " + device_type + " is not supported with precision suffix. \n");
}
const auto& valid_values = helper[device_prefix].second;
if (std::find(std::begin(valid_values), std::end(valid_values), precision) != std::end(valid_values)) {
return precision;
} else {
if (helper.contains(device_type)) {
auto const& valid_values = helper[device_type].second;
auto value_iter = valid_values.begin();
std::string valid_values_joined = *value_iter;
// Append 2nd and up, if only one then ++value_iter is same as end()
for (++value_iter; value_iter != valid_values.end(); ++value_iter) {
valid_values_joined += ", " + *value_iter;
}

if (precision == "ACCURACY") {
return valid_values.back(); // Return highest supported precision
} else {
if (std::find(valid_values.begin(), valid_values.end(), precision) != valid_values.end()) {
return precision; // Return precision selected if valid
} else {
auto value_iter = valid_values.begin();
std::string valid_values_joined = *value_iter;
// Append 2nd and up, if only one then ++value_iter is same as end()
for (++value_iter; value_iter != valid_values.end(); ++value_iter) {
valid_values_joined += ", " + *value_iter;
}
ORT_THROW("[ERROR] [OpenVINO] Unsupported inference precision is selected. ", device_type, " only supports ", valid_values_joined, ".\n");
}
}

ORT_THROW("[ERROR] [OpenVINO] Unsupported inference precision is selected. ",
device_type, " only supports", valid_values_joined, ".\n");
}
}
} else if (deprecated_device_types.contains(device_type)) {
LOGS_DEFAULT(WARNING)
<< "[OpenVINO] Selected 'device_type' " + device_type + " is deprecated. \n"
<< "Update the 'device_type' to specified types 'CPU', 'GPU', 'GPU.0', "
<< "'GPU.1', 'NPU' or from HETERO/MULTI/AUTO options and set 'precision' separately. \n";
auto delimit = device_type.find("_");
device_type = device_type.substr(0, delimit);
return device_type.substr(delimit + 1);
// Deprecated devices are already handled above
// We have to remove the index (.0, .1, etc.) to use device as key for helper
auto device_prefix = device_type;
if (auto dot_delimit = device_prefix.find("."); dot_delimit != std::string::npos) {
device_prefix = device_prefix.substr(0, dot_delimit);
}

if (provider_options.contains(option_name)) {
std::string precision = provider_options.at(option_name);

if (helper.contains(device_prefix)) {
auto const& valid_values = helper[device_prefix].second;
if (std::find(std::begin(valid_values), std::end(valid_values), precision) != std::end(valid_values)) {
return precision; // Return precision selected if valid
} else {
ORT_THROW("[ERROR] [OpenVINO] Unsupported device type provided: ",
device_type, "\n");
auto value_iter = valid_values.begin();
std::string valid_values_joined = *value_iter;

Check warning on line 94 in onnxruntime/core/providers/openvino/openvino_parser_utils.cc

View workflow job for this annotation

GitHub Actions / Optional Lint C++ 

[cpplint] reported by reviewdog 🐶
 Add #include <string> for string  [build/include_what_you_use] [4]

Raw Output:
onnxruntime/core/providers/openvino/openvino_parser_utils.cc:94:  Add #include <string> for string  [build/include_what_you_use] [4]
// Append 2nd and up, if only one then ++value_iter is same as end()
for (++value_iter; value_iter != valid_values.end(); ++value_iter) {
valid_values_joined += ", " + *value_iter;
}

ORT_THROW("[ERROR] [OpenVINO] Unsupported inference precision is selected. ", device_type, " only supports ", valid_values_joined, ".\n");
}
} else {
// Not found in helper - custom device, return as is
return precision;
}
} else {
if (device_type.find("NPU") != std::string::npos || device_type.find("GPU") != std::string::npos) {
return "FP16";
} else if (device_type.find("CPU") != std::string::npos) {
return "FP32";
// Precision not set
if (helper.contains(device_prefix)) {
// If found in helper - set the default
return helper[device_prefix].first;
} else {
ORT_THROW("[ERROR] [OpenVINO] Unsupported device is selected", device_type, "\n");
// Not found in helper - custom device - default precision
LOGS_DEFAULT(INFO) << "[OpenVINO] Precision is not set. Using default OpenVINO precision for " + device_type + ". \n";
return "";
}
}
}
Expand Down
107 changes: 62 additions & 45 deletions onnxruntime/core/providers/openvino/openvino_provider_factory.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,63 +57,80 @@
}

std::string ParseDeviceType(std::shared_ptr<OVCore> ov_core, const ProviderOptions& provider_options, std::string option_name) {
const std::vector<std::string> ov_available_devices = ov_core->GetAvailableDevices();

std::set<std::string> ov_supported_device_types = {"CPU", "GPU",
"GPU.0", "GPU.1", "NPU"};
std::set<std::string> deprecated_device_types = {"CPU_FP32", "GPU_FP32",
"GPU.0_FP32", "GPU.1_FP32", "GPU_FP16",
"GPU.0_FP16", "GPU.1_FP16"};

// Expand set of supported device with OV devices
ov_supported_device_types.insert(ov_available_devices.begin(), ov_available_devices.end());
// This function normally does not check if the selected device is available, but does some sanity checks
// Only if the device is not standard, then availability is checked.
// Availability is checked for the selected device in the OpenVINOExecutionProvider constructor

std::vector<std::string> devices_to_check;
std::string selected_device;
if (provider_options.contains(option_name)) {
const auto& selected_device = provider_options.at("device_type");

if (deprecated_device_types.contains(selected_device)) {
// Deprecated device and precision is handled together at ParsePrecision
return selected_device;
}

if (!((ov_supported_device_types.contains(selected_device)) ||
(selected_device.find("HETERO:") == 0) ||
(selected_device.find("MULTI:") == 0) ||
(selected_device.find("AUTO:") == 0))) {
ORT_THROW(
"[ERROR] [OpenVINO] You have selected wrong configuration value for the key 'device_type'. "
"Select from 'CPU', 'GPU', 'NPU', 'GPU.x' where x = 0,1,2 and so on or from"
" HETERO/MULTI/AUTO options available. \n");
selected_device = provider_options.at(option_name);
// If we have multiple device configuration, we need to check all of them
if ((selected_device.find("HETERO:") == 0) ||
(selected_device.find("MULTI:") == 0) ||
(selected_device.find("BATCH:") == 0) ||
(selected_device.find("AUTO:") == 0)) {
auto delimit = selected_device.find(":");
const auto& devices = selected_device.substr(delimit + 1);
devices_to_check = split(devices, ',');
} else {
devices_to_check.push_back(selected_device);
}
LOGS_DEFAULT(INFO) << "[OpenVINO-EP] Choosing Device: " << selected_device;
return selected_device;
} else {
std::string default_device;

// Take default behavior from project configuration
#if defined OPENVINO_CONFIG_CPU
default_device = "CPU";
selected_device = "CPU";
LOGS_DEFAULT(INFO) << "[OpenVINO-EP] Choosing Device: " << selected_device;
return selected_device;
#elif defined OPENVINO_CONFIG_GPU
default_device = "GPU";
selected_device = "GPU";
LOGS_DEFAULT(INFO) << "[OpenVINO-EP] Choosing Device: " << selected_device;
return selected_device;
#elif defined OPENVINO_CONFIG_NPU
default_device = "NPU";
selected_device = "NPU";
LOGS_DEFAULT(INFO) << "[OpenVINO-EP] Choosing Device: " << selected_device;
return selected_device;
#elif defined OPENVINO_CONFIG_HETERO || defined OPENVINO_CONFIG_MULTI || defined OPENVINO_CONFIG_AUTO
default_device = DEVICE_NAME;

// Validate that devices passed are valid
int delimit = device_type.find(":");
const auto& devices = device_type.substr(delimit + 1);
auto device_list = split(devices, ',');
for (const auto& device : devices) {
if (!ov_supported_device_types.contains(device)) {
ORT_THROW("[ERROR] [OpenVINO] Invalid device selected: ", device);
}
}
selected_device = DEVICE_NAME;

// Add sub-devices to check-list
int delimit = selected_device.find(":");
const auto& devices = selected_device.substr(delimit + 1);
devices_to_check = split(devices, ',');
#endif
}

LOGS_DEFAULT(INFO) << "[OpenVINO-EP] Choosing Device: " << default_device;
return default_device;
// Devices considered to be supported by default
std::unordered_set<std::string> supported_device_types = {"CPU", "GPU", "NPU"};

Check warning on line 104 in onnxruntime/core/providers/openvino/openvino_provider_factory.cc

View workflow job for this annotation

GitHub Actions / Optional Lint C++ 

[cpplint] reported by reviewdog 🐶
 Add #include <unordered_set> for unordered_set<>  [build/include_what_you_use] [4]

Raw Output:
onnxruntime/core/providers/openvino/openvino_provider_factory.cc:104:  Add #include <unordered_set> for unordered_set<>  [build/include_what_you_use] [4]
for (auto device : devices_to_check) {
// Check deprecated device format (CPU_FP32, GPU.0_FP16, etc.) and remove the suffix in place
// Suffix will be parsed in ParsePrecision
if (auto delimit = device.find("_"); delimit != std::string::npos) {
device = device.substr(0, delimit);
}
// Just the device name without .0, .1, etc. suffix
auto device_prefix = device;
// Check if device index is appended (.0, .1, etc.), if so, remove it
if (auto delimit = device_prefix.find("."); delimit != std::string::npos) {
device_prefix = device_prefix.substr(0, delimit);
if (device_prefix == "CPU")
ORT_THROW("[ERROR] [OpenVINO] CPU device is only supported without index, CPU.x is illegal.\n");
}
// Only device is not supported by default (some exotic device), check if it's available
if (!supported_device_types.contains(device_prefix)) {
std::vector<std::string> available_devices = ov_core->GetAvailableDevices();
// Here we need to find the full device name (with .idx, but without _precision)
if (std::find(std::begin(available_devices), std::end(available_devices), device) == std::end(available_devices)) {
ORT_THROW(
"[ERROR] [OpenVINO] You have selected wrong configuration value for the key 'device_type'. "
"Select from 'CPU', 'GPU', 'NPU', 'GPU.x' where x = 0,1,2 and so on or from"
" HETERO/MULTI/AUTO/BATCH options available. \n");
}
}
}
// All devices have passed the check, return selected device
LOGS_DEFAULT(INFO) << "[OpenVINO-EP] Choosing Device: " << selected_device;
return selected_device;
}

void ParseProviderOptions([[maybe_unused]] ProviderInfo& result, [[maybe_unused]] const ProviderOptions& config_options) {}
Expand Down
Loading