Testing

Testing#

HIP-VS has its own unit testing for both targets AMD via the clang compiler and NVIDIA via the nvcc compiler. Testing is independent of the physical hardware. That means that physical GPU(s) might be from any vendor; discrete GPU might be absent at all.

Test dependencies:

For testing the HIP-VS extension, the dependencies are the same as for the HIP-VS extension itself.

  • For clang tests: AMD HIP SDK 6.2 should be installed, and the environment variable HIP_PATH should be correctly set to its root.

  • For nvcc tests: CUDA Toolkit 12.5.1 for Windows should be installed, and the environment variable CUDA_PATH should be correctly set to its root.

Both CUDA and HIP SDK can be installed on the same system and tested by HIP-VS unit tests.

Before testing, the target HIP-VS solution should be built, and the corresponding vsix should be installed into the target Visual Studio successfully.

Test source declarations:

  • For clang tests: extension\HipExtension.Tests.Clang\testdata.json.

  • For nvcc tests: extension\HipExtension.Tests.Nvcc\testdata.json.

Test build logs and the compiled binaries:

  • For clang tests: extension\HipExtension.Tests.Clang\bin\Release\tests.

  • For nvcc tests: extension\HipExtension.Tests.Nvcc\bin\Release\tests.

Testing in Visual Studio IDE 2022, 2019, or 2017#

1. If the HIP-VS extension was installed previously for the Visual Studio, it should be uninstalled first, and separately for each Studio in case of multiple Visual Studios installed.

2. Perform building of the HIPExtension2022.sln, HIPExtension2019.sln, or HIPExtension2017.sln solution depending on the desired target Studio.

3. Close all the running instances of Microsoft Visual Studio.

4. Install the HIP-VS extension by running HIPExtension2022.vsix, HIPExtension2019.vsix, or HIPExtension2017.vsix correspondingly.

5. Open the HIPExtension2022.sln, HIPExtension2019.sln, or HIPExtension2017.sln solution once again.

6. Select Test -> Test Explorer in VS 2022 or VS 2019:

image

or Test -> Windows -> Test Explorer in VS 2017:

image

7. The following Test Explorer main window should appear:

For VS 2022:

image

For VS 2019:

image

For VS 2017:

image

On the left side, there is a test tree with two top nodes with test collections for clang and nvcc correspondingly.

8. Expand one of the top nodes: HIPExtension.Tests.Clang.20XX* (from here screenshots are provided only for VS 2022 due to their similarity for all target Visual Studios):

image

or HIPExtension.Tests.Nvcc.20XX*:

image

* XX - 17, 19, or 22 correspondingly.

The following test collections are available for both clang and nvcc:

  • Options - the tests for the options which are set using HIP-VS’s Platform Toolset AMD HIP for clang Compiler in the HIP project Property Pages:

    • Devenv - option tests to run via devenv.exe;

    • MSBuild - the same option tests to run via msbuild.exe;

    • Options.Parallel_XX.Devenv.VSYY - the same option tests to run them in parallel via devenv.exe, where XX - number of threads (02, 04, 06, 08, 12, 16, 24, 32), and YY - version of Visual Studio to run tests against (15 = 2017, 16 = 2019, 17 = 2022)**;

    • Options.Parallel_XX.MSBuild.VSYY - the same option tests to run them in parallel via msbuild.exe;

  • Multifile.BuildTests - tests for building projects with multiple HIP source files;

  • IncrementalBuild*** - tests for the incremental building mode; the tests prove that only the changed header or source file was actually recompiled:

    • HeaderTests - tests in which one of the dependent header files is changed;

    • OuterHeaderTests - tests in which one of the dependent, but not presented in the project, header files are changed;

    • SourceTests - tests in which one of the source files is changed;

  • TemplateProjects.BuildTests - tests for building executables from the MatrixTranspose sample template with the default options and settings.

** For testing in HIPExtension2022.sln solution only VS17 tests are valid.

** For testing in HIPExtension2019.sln solution only VS16 tests are valid.

** For testing in HIPExtension2017.sln solution only VS15 tests are valid.

** For testing in HIPExtension.sln solution (if opened in VS 2019 only) all VS17, VS17 Preview, VS16, and VS15 tests are valid.****

*** Incremental building has not been implemented for the nvcc path yet.

**** HIPExtension.sln is only for HIP-VS developing purposes in multi-target mode; it is not recommended to use it, all the target Visual Studios should be installed, the procedure of building and installing for all the Visual Studios differs.

9. For all test collections, except parallel, the corresponding VSYY subset should be chosen to run in the drop-down list, for instance:

image

Testing in shell#

Consider HIP-VS as a root folder of the cloned HIP-VS project:

set HIP-VS="<path_to_the_root_folder_of_the_cloned_HIP-VS>"

or

setx HIP-VS "<path_to_the_root_folder_of_the_cloned_HIP-VS>"

Testing in shell for VS 2017#

1. Change the current directory to the corresponding subfolder with clang tests:

cd %HIP-VS%\extension\HipExtension.Tests.Clang\bin\Release

or nvcc tests:

cd %HIP-VS%\extension\HipExtension.Tests.Nvcc\bin\Release

2. Run testing in a single-threaded mode for Options tests via MSBuild targeting clang:

"c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Clang.2017.dll /TestCaseFilter:"FullyQualifiedName~Options.MSBuild.VS15" /Parallel

or nvcc:

"c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Nvcc.2017.dll /TestCaseFilter:"FullyQualifiedName~Options.MSBuild.VS15" /Parallel

3. Run testing in a single-threaded mode for Options tests via Devenv targeting clang:

"c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Clang.2017.dll /TestCaseFilter:"FullyQualifiedName~Options.Devenv.VS15" /Parallel

or nvcc:

"c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Nvcc.2017.dll /TestCaseFilter:"FullyQualifiedName~Options.Devenv.VS15" /Parallel

4. Run testing in a multi-threaded mode for Options tests via Devenv targeting clang:

"c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Clang.2017.dll /TestCaseFilter:"FullyQualifiedName~Options.Parallel_XX.Devenv.VS15" /Parallel

or nvcc:

"c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Nvcc.2017.dll /TestCaseFilter:"FullyQualifiedName~Options.Parallel_XX.Devenv.VS15" /Parallel

* XX - the number of threads (02, 04, 06, 08, 12, 16, 24, 32).

Testing in shell for VS 2019#

1. Change the current directory to the corresponding subfolder with clang tests:

cd %HIP-VS%\extension\HipExtension.Tests.Clang\bin\Release

or nvcc tests:

cd %HIP-VS%\extension\HipExtension.Tests.Nvcc\bin\Release

2. Run testing in a single-threaded mode for Options tests via MSBuild targeting clang:

"c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Clang.2019.dll /TestCaseFilter:"FullyQualifiedName~Options.MSBuild.VS16" /Parallel

or nvcc:

"c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Nvcc.2019.dll /TestCaseFilter:"FullyQualifiedName~Options.MSBuild.VS16" /Parallel

3. Run testing in a single-threaded mode for Options tests via Devenv targeting clang:

"c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Clang.2019.dll /TestCaseFilter:"FullyQualifiedName~Options.Devenv.VS16" /Parallel

or nvcc:

"c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Nvcc.2019.dll /TestCaseFilter:"FullyQualifiedName~Options.Devenv.VS16" /Parallel

4. Run testing in a multi-threaded mode for Options tests via Devenv targeting clang:

"c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Clang.2019.dll /TestCaseFilter:"FullyQualifiedName~Options.Parallel_XX.Devenv.VS16" /Parallel

or nvcc:

"c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Nvcc.2019.dll /TestCaseFilter:"FullyQualifiedName~Options.Parallel_XX.Devenv.VS16" /Parallel

* XX - the number of threads (02, 04, 06, 08, 12, 16, 24, 32).

Testing in shell for VS 2022#

1. Change the current directory to the corresponding subfolder with clang tests:

cd %HIP-VS%\extension\HipExtension.Tests.Clang\bin\Release

or nvcc tests:

cd %HIP-VS%\extension\HipExtension.Tests.Nvcc\bin\Release

2. Run testing in a single-threaded mode for Options tests via MSBuild targeting clang:

"c:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Clang.2022.dll /TestCaseFilter:"FullyQualifiedName=Options.MSBuild.VS17" /Parallel

or nvcc:

"c:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Nvcc.2022.dll /TestCaseFilter:"FullyQualifiedName=Options.MSBuild.VS17" /Parallel

3. Run testing in a single-threaded mode for Options tests via Devenv targeting clang:

"c:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Clang.2022.dll /TestCaseFilter:"FullyQualifiedName=Options.Devenv.VS17" /Parallel

or nvcc:

"c:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Nvcc.2022.dll /TestCaseFilter:"FullyQualifiedName=Options.Devenv.VS17" /Parallel

4. Run testing in a multi-threaded mode for Options tests via Devenv targeting clang:

"c:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Clang.2022.dll /TestCaseFilter:"FullyQualifiedName~Options.Parallel_XX.Devenv.VS17" /Parallel

or nvcc:

"c:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" HipExtension.Tests.Nvcc.2022.dll /TestCaseFilter:"FullyQualifiedName~Options.Parallel_XX.Devenv.VS17" /Parallel

* XX - the number of threads (02, 04, 06, 08, 12, 16, 24, 32).