diff --git a/docs/api/Inside-the-Template-Engine.md b/docs/api/Inside-the-Template-Engine.md index fe72a5cc967..8a2f2e1b183 100644 --- a/docs/api/Inside-the-Template-Engine.md +++ b/docs/api/Inside-the-Template-Engine.md @@ -18,7 +18,7 @@ designed to separate gathering, instantiating and processing templates. # Overview of the subsystems -## [IEngineEnvironmentSettings](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Abstractions/IEngineEnvironmentSettings.cs) +## [IEngineEnvironmentSettings](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Abstractions/IEngineEnvironmentSettings.cs) This is responsible for holding all properties of the environment. @@ -69,7 +69,7 @@ and `ComponentManager` is responsible for loading [components](#components) provided by the host from `ITemplateEngineHost.BuiltIns`. Additional components can be added via -`IComponentManager.AddComponent`. Similar to templates, components are also +`IComponentManager.AddComponent`. Similar to templates, components are also dynamically loaded from template packages during scanning. `ComponentManager` implementation is not publicly accessible; however, it is used when default implementation of `IEngineEnvironmentSettings` is used. @@ -120,9 +120,9 @@ instantiate the class when needed. ### Supplying parameters conditions results -It is possible to supply evaluation results of parameters conditions when instantiating template via Edge API [`TemplateCreator.InstantiateAsync`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/TemplateCreator.cs#L89). Example use case is instantiation from Visual Studio host, that will leverage condition evaluator integrated within the New Project Dialog. +It is possible to supply evaluation results of parameters conditions when instantiating template via Edge API [`TemplateCreator.InstantiateAsync`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/TemplateCreator.cs#L89). Example use case is instantiation from Visual Studio host, that will leverage condition evaluator integrated within the New Project Dialog. -This can be achieved by passing the structured [`InputDataSet`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/InputDataSet.cs) argument that is populated with [`EvaluatedInputParameterData`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/EvaluatedInputParameterData.cs) objects for evaluated parameters. +This can be achieved by passing the structured [`InputDataSet`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/InputDataSet.cs) argument that is populated with [`EvaluatedInputParameterData`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/EvaluatedInputParameterData.cs) objects for evaluated parameters. It is currently not possible to provide just partial external evaluation - meaning that the template engine evaluates either all the parameter conditions or none. If the [`InputDataSet`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/InputDataSet.cs) collection contains at least one [`EvaluatedInputParameterData`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/EvaluatedInputParameterData.cs) element, results of all parameter conditions are expected to be passed. @@ -178,7 +178,7 @@ be split into two main categories: - mount points -## Template package providers +## Template package providers Template package providers responsible for providing template packages the template engine needs to manage. There are two kinds of providers: @@ -243,6 +243,12 @@ Template engine provides two installer implementations: You can use them when implementing your own providers, they are available from IComponentManager. +When package verification is enabled, downloaded signed NuGet packages are +verified using `dotnet nuget verify --all` before installation. Unsigned packages +are accepted only when NuGet's `signatureValidationMode` is `accept` and the +source does not declare `allRepositorySigned=true`. A client policy of `require` +or a repository signing requirement rejects unsigned packages. + ## [Mount point](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Abstractions/Mount/IMountPoint.cs) Template package location is represented by mount point URI. Mount point that @@ -268,14 +274,14 @@ Template engine host works with constraint using `TemplateConstraintManager` cla The template may define `bind` symbols. Those symbols might be bound to values provided externally using bind symbol source components. Default sources are: -- host provided parameters - uses prefix `host:`. Binds symbol to parameters provided by the host. +- host provided parameters - uses prefix `host:`. Binds symbol to parameters provided by the host. - environment variables - uses prefix `env:`. Binds symbol to environment variables. These components are defined in `Microsoft.TemplateEngine.Edge` and are part of mandatory components collection to be loaded. -If the `binding` does not have prefix, or prefixed value cannot be evaluated the evaluation without prefix is performed as the fallback. +If the `binding` does not have prefix, or prefixed value cannot be evaluated the evaluation without prefix is performed as the fallback. In case there are more than one component which returns the value, the component priority will be used to define the value to be bounded. In case of same priorities, the evaluation results in error. -The template author needs to explicitly specify the prefix in this case. It is possible to restrict usage of prefix with setting `IBindSymbolSource.RequiresPrefixMatch` to `true`. +The template author needs to explicitly specify the prefix in this case. It is possible to restrict usage of prefix with setting `IBindSymbolSource.RequiresPrefixMatch` to `true`. ## Registering the components @@ -325,7 +331,7 @@ It is possible to register additional components to in the following way: Note that components are not persisted, so they should be added each time EngineEnvironmentSettings instance is created. -# Microsoft.TemplateEngine.IDE +# Microsoft.TemplateEngine.IDE The package provides [Bootstrapper](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.IDE/Bootstrapper.cs) diff --git a/eng/Version.Details.props b/eng/Version.Details.props index 79b0103b143..4cedaa86630 100644 --- a/eng/Version.Details.props +++ b/eng/Version.Details.props @@ -7,12 +7,12 @@ This file should be imported by eng/Versions.props 10.0.11 - 10.0.0-beta.26512.108 + 10.0.0-beta.26563.104 10.0.11 10.0.11 10.0.11 10.0.11 - 2.0.14 + 2.0.15 10.0.11 10.0.11 10.0.11 diff --git a/eng/Version.Details.xml b/eng/Version.Details.xml index 23819df5508..8c9c4d7dcdc 100644 --- a/eng/Version.Details.xml +++ b/eng/Version.Details.xml @@ -1,16 +1,16 @@ - + - - https://github.com/dotnet/dotnet - f009a9309c13506b779f3163c50c61381627a1e6 + + https://dev.azure.com/dnceng/internal/_git/dotnet-dotnet + 8e0f8ed2312954727f8919d7d1c2f78b483b4ee7 - + https://github.com/dotnet/dotnet - f009a9309c13506b779f3163c50c61381627a1e6 + d2b8c5ed199ecb1f3bec494b5d1b7ee84789ec38 diff --git a/eng/Versions.props b/eng/Versions.props index 57147b9694c..8063efb0384 100644 --- a/eng/Versions.props +++ b/eng/Versions.props @@ -1,7 +1,7 @@ - 10.0.403 + 10.0.405 true diff --git a/eng/common/core-templates/job/publish-build-assets.yml b/eng/common/core-templates/job/publish-build-assets.yml index 718b7f0a0ac..73468b8e240 100644 --- a/eng/common/core-templates/job/publish-build-assets.yml +++ b/eng/common/core-templates/job/publish-build-assets.yml @@ -187,10 +187,10 @@ jobs: PromoteToChannelIds: ${{ parameters.PromoteToChannelIds }} is1ESPipeline: ${{ parameters.is1ESPipeline }} - # Darc is targeting 8.0, so make sure it's installed + # Darc is targeting 10.0, so make sure it's installed - task: UseDotNet@2 inputs: - version: 8.0.x + version: 10.0.x - task: AzureCLI@2 displayName: Publish Using Darc diff --git a/eng/common/core-templates/post-build/post-build.yml b/eng/common/core-templates/post-build/post-build.yml index 135fc9a5051..fbdd0eadc66 100644 --- a/eng/common/core-templates/post-build/post-build.yml +++ b/eng/common/core-templates/post-build/post-build.yml @@ -355,10 +355,10 @@ stages: - template: /eng/common/templates/steps/enable-internal-runtimes.yml - # Darc is targeting 8.0, so make sure it's installed + # Darc is targeting 10.0, so make sure it's installed - task: UseDotNet@2 inputs: - version: 8.0.x + version: 10.0.x - task: AzureCLI@2 displayName: Publish Using Darc diff --git a/eng/common/tools.ps1 b/eng/common/tools.ps1 index 295f26c351c..c8b1f9d2566 100644 --- a/eng/common/tools.ps1 +++ b/eng/common/tools.ps1 @@ -35,7 +35,8 @@ [bool]$warnAsError = if (Test-Path variable:warnAsError) { $warnAsError } else { $true } # Specifies semi-colon delimited list of warning codes that should not be treated as errors. -[string]$warnNotAsError = if (Test-Path variable:warnNotAsError) { $warnNotAsError } else { '' } +# Defaults to NuGet Audit warning codes NU1901-NU1904. +[string]$warnNotAsError = if ((Test-Path variable:warnNotAsError) -and $warnNotAsError) { $warnNotAsError } else { 'NU1901;NU1902;NU1903;NU1904' } # Specifies which msbuild engine to use for build: 'vs', 'dotnet' or unspecified (determined based on presence of tools.vs in global.json). [string]$msbuildEngine = if (Test-Path variable:msbuildEngine) { $msbuildEngine } else { $null } diff --git a/eng/common/tools.sh b/eng/common/tools.sh index df76f062a76..28ec1a491e4 100755 --- a/eng/common/tools.sh +++ b/eng/common/tools.sh @@ -53,7 +53,8 @@ fi warn_as_error=${warn_as_error:-true} # Specifies semi-colon delimited list of warning codes that should not be treated as errors. -warn_not_as_error=${warn_not_as_error:-''} +# Defaults to NuGet Audit warning codes NU1901-NU1904. +warn_not_as_error="${warn_not_as_error:-NU1901;NU1902;NU1903;NU1904}" # True to attempt using .NET Core already that meets requirements specified in global.json # installed on the machine instead of downloading one. diff --git a/global.json b/global.json index d76ccf85714..671fbbfaa94 100644 --- a/global.json +++ b/global.json @@ -1,6 +1,6 @@ { "sdk": { - "version": "10.0.401", + "version": "10.0.404", "allowPrerelease": true, "rollForward": "latestFeature", "paths": [ @@ -10,9 +10,9 @@ "errorMessage": "The required .NET SDK wasn't found. Please run ./eng/common/dotnet.cmd/sh to install it." }, "tools": { - "dotnet": "10.0.401" + "dotnet": "10.0.404" }, "msbuild-sdks": { - "Microsoft.DotNet.Arcade.Sdk": "10.0.0-beta.26512.108" + "Microsoft.DotNet.Arcade.Sdk": "10.0.0-beta.26563.104" } } diff --git a/src/Microsoft.TemplateEngine.Edge/Installers/NuGet/DotnetNuGetPackageVerifier.cs b/src/Microsoft.TemplateEngine.Edge/Installers/NuGet/DotnetNuGetPackageVerifier.cs new file mode 100644 index 00000000000..dc9dd6966d9 --- /dev/null +++ b/src/Microsoft.TemplateEngine.Edge/Installers/NuGet/DotnetNuGetPackageVerifier.cs @@ -0,0 +1,164 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System.ComponentModel; +using System.Diagnostics; +using System.Runtime.InteropServices; +using System.Security; +using Microsoft.TemplateEngine.CommandUtils; +using NuGet.Common; +using NuGet.Configuration; +using NuGet.Packaging; +using NuGet.Packaging.Signing; +using NuGet.Protocol; +using NuGet.Protocol.Core.Types; + +namespace Microsoft.TemplateEngine.Edge.Installers.NuGet +{ + internal interface INuGetPackageVerifier + { + bool IsEnabled { get; } + + Task VerifyAsync(string packagePath, SourceRepository repository, ISettings settings, ILogger logger, CancellationToken cancellationToken); + } + + internal sealed class DotnetNuGetPackageVerifier : INuGetPackageVerifier + { + private const string SignatureVerificationEnvironmentVariable = "DOTNET_NUGET_SIGNATURE_VERIFICATION"; + + public DotnetNuGetPackageVerifier() + { + bool? configuredValue = bool.TryParse( + Environment.GetEnvironmentVariable(SignatureVerificationEnvironmentVariable), + out bool shouldVerify) + ? shouldVerify + : null; + + IsEnabled = RuntimeInformation.IsOSPlatform(OSPlatform.Windows) + || configuredValue == true + || (configuredValue == null && RuntimeInformation.IsOSPlatform(OSPlatform.Linux)); + } + + public bool IsEnabled { get; } + + public async Task VerifyAsync(string packagePath, SourceRepository repository, ISettings settings, ILogger logger, CancellationToken cancellationToken) + { + _ = packagePath ?? throw new ArgumentNullException(nameof(packagePath)); + _ = repository ?? throw new ArgumentNullException(nameof(repository)); + _ = settings ?? throw new ArgumentNullException(nameof(settings)); + _ = logger ?? throw new ArgumentNullException(nameof(logger)); + cancellationToken.ThrowIfCancellationRequested(); + + bool isSigned; + using (var reader = new PackageArchiveReader(packagePath)) + { + isSigned = await reader.IsSignedAsync(cancellationToken).ConfigureAwait(false); + } + + if (!isSigned) + { + ClientPolicyContext clientPolicy = ClientPolicyContext.GetClientPolicy(settings, logger); + RepositorySignatureResource? repositorySignature = + await repository.GetResourceAsync(cancellationToken).ConfigureAwait(false); + cancellationToken.ThrowIfCancellationRequested(); + if (clientPolicy.Policy == SignatureValidationMode.Require || repositorySignature?.AllRepositorySigned == true) + { + const string message = "The package is unsigned, but client or repository policy requires signing."; + logger.LogDebug(message); + throw new InvalidDataException(message); + } + + logger.LogDebug("Accepting unsigned package under accept policy."); + return; + } + + await VerifySignedPackageAsync(packagePath, logger, cancellationToken).ConfigureAwait(false); + } + + private static async Task VerifySignedPackageAsync(string packagePath, ILogger logger, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + using var process = new Process + { + StartInfo = new ProcessStartInfo + { + FileName = "dotnet", + Arguments = $"nuget verify --all {ArgumentEscaper.EscapeSingleArg(packagePath)}", + UseShellExecute = false, + CreateNoWindow = true, + RedirectStandardError = true, + RedirectStandardOutput = true + } + }; + if (!process.Start()) + { + throw new InvalidOperationException("Failed to start 'dotnet nuget verify'."); + } + +#if NET + Task standardOutput = process.StandardOutput.ReadToEndAsync(CancellationToken.None); + Task standardError = process.StandardError.ReadToEndAsync(CancellationToken.None); +#else + Task standardOutput = process.StandardOutput.ReadToEndAsync(); + Task standardError = process.StandardError.ReadToEndAsync(); +#endif + await WaitForExitAsync(process, logger, cancellationToken).ConfigureAwait(false); + + string output = await standardOutput.ConfigureAwait(false); + string error = await standardError.ConfigureAwait(false); + cancellationToken.ThrowIfCancellationRequested(); + + LogOutput(logger, output, LogLevel.Debug); + LogOutput(logger, error, process.ExitCode == 0 ? LogLevel.Debug : LogLevel.Error); + + if (process.ExitCode != 0) + { + string command = $"{process.StartInfo.FileName} {process.StartInfo.Arguments}"; + throw new SecurityException($"'{command}' failed with exit code {process.ExitCode}."); + } + + return; + + static void LogOutput(ILogger logger, string output, LogLevel level) + { + if (!string.IsNullOrWhiteSpace(output)) + { + logger.Log(level, output.Trim()); + } + } + } + + private static async Task WaitForExitAsync(Process process, ILogger logger, CancellationToken cancellationToken) + { + using (cancellationToken.Register(() => TerminateProcess(process, logger))) + { + // Process.WaitForExitAsync is unavailable on netstandard2.0 and net472. + await Task.Run(process.WaitForExit, CancellationToken.None).ConfigureAwait(false); + } + + return; + + static void TerminateProcess(Process process, ILogger logger) + { + try + { + if (!process.HasExited) + { + process.Kill(); + } + } + catch (InvalidOperationException) when (process.HasExited) + { + // The process exited between checking HasExited and calling Kill. + } + catch (Win32Exception ex) + { + if (!process.HasExited) + { + logger.LogDebug($"Failed to terminate 'dotnet nuget verify': {ex}"); + } + } + } + } + } +} diff --git a/src/Microsoft.TemplateEngine.Edge/Installers/NuGet/NugetApiPackageManager.cs b/src/Microsoft.TemplateEngine.Edge/Installers/NuGet/NugetApiPackageManager.cs index ba4d3d22ccb..3c3cf1352d1 100644 --- a/src/Microsoft.TemplateEngine.Edge/Installers/NuGet/NugetApiPackageManager.cs +++ b/src/Microsoft.TemplateEngine.Edge/Installers/NuGet/NugetApiPackageManager.cs @@ -10,6 +10,7 @@ using NuGet.Protocol.Core.Types; using NuGet.Versioning; using ILogger = NuGet.Common.ILogger; +using NuGetSettings = NuGet.Configuration.Settings; namespace Microsoft.TemplateEngine.Edge.Installers.NuGet { @@ -18,6 +19,7 @@ internal class NuGetApiPackageManager : IDownloader, IUpdateChecker private static readonly ConcurrentDictionary SourcesCache = new(); private readonly IEngineEnvironmentSettings _environmentSettings; private readonly ILogger _nugetLogger; + private readonly INuGetPackageVerifier _packageVerifier; private readonly SourceCacheContext _cacheSettings = new SourceCacheContext() { @@ -25,10 +27,11 @@ internal class NuGetApiPackageManager : IDownloader, IUpdateChecker DirectDownload = true }; - internal NuGetApiPackageManager(IEngineEnvironmentSettings settings) + internal NuGetApiPackageManager(IEngineEnvironmentSettings settings, INuGetPackageVerifier? packageVerifier = null) { _environmentSettings = settings ?? throw new ArgumentNullException(nameof(settings)); _nugetLogger = new NuGetLogger(_environmentSettings.Host.LoggerFactory); + _packageVerifier = packageVerifier ?? new DotnetNuGetPackageVerifier(); } /// @@ -111,50 +114,46 @@ await GetLatestVersionInternalAsync( _nugetLogger.LogError(string.Format(LocalizableStrings.NuGetApiPackageManager_Error_FileAlreadyExists, filePath)); throw new DownloadException(packageMetadata.Identity.Id, packageMetadata.Identity.Version.ToNormalizedString(), new[] { source.Source }); } + + string temporaryFilePath = filePath + "." + Guid.NewGuid().ToString("N") + ".tmp"; try { - using Stream packageStream = _environmentSettings.Host.FileSystem.CreateFile(filePath); - if (await resource.CopyNupkgToStreamAsync( - packageMetadata.Identity.Id, - packageMetadata.Identity.Version, - packageStream, - _cacheSettings, - _nugetLogger, - cancellationToken).ConfigureAwait(false)) + bool downloadSucceeded; + using (Stream packageStream = _environmentSettings.Host.FileSystem.CreateFile(temporaryFilePath)) { - return new NuGetPackageInfo( - packageMetadata.Authors, - packageMetadata.Owners, - reserved: packageMetadata.PrefixReserved, - filePath, - source.Source, + downloadSucceeded = await resource.CopyNupkgToStreamAsync( packageMetadata.Identity.Id, - packageMetadata.Identity.Version.ToNormalizedString(), - packageMetadata.Vulnerabilities); + packageMetadata.Identity.Version, + packageStream, + _cacheSettings, + _nugetLogger, + cancellationToken).ConfigureAwait(false); } - else + + if (!downloadSucceeded) { _nugetLogger.LogWarning( string.Format( LocalizableStrings.NuGetApiPackageManager_Warning_FailedToDownload, $"{packageMetadata.Identity.Id}@{packageMetadata.Identity.Version}", source.Source)); - try - { - _environmentSettings.Host.FileSystem.FileDelete(filePath); - } - catch (Exception ex) - { - _nugetLogger.LogWarning( - string.Format( - LocalizableStrings.NuGetApiPackageManager_Warning_FailedToDelete, - filePath)); - _nugetLogger.LogDebug($"Details: {ex}."); - } throw new DownloadException(packageMetadata.Identity.Id, packageMetadata.Identity.Version.ToNormalizedString(), new[] { source.Source }); } + + await VerifyPackageAsync(temporaryFilePath, cancellationToken).ConfigureAwait(false); + TryCopyFile(temporaryFilePath, filePath); + + return new NuGetPackageInfo( + packageMetadata.Authors, + packageMetadata.Owners, + reserved: packageMetadata.PrefixReserved, + filePath, + source.Source, + packageMetadata.Identity.Id, + packageMetadata.Identity.Version.ToNormalizedString(), + packageMetadata.Vulnerabilities); } - catch (Exception e) + catch (Exception e) when (e is not DownloadException) { _nugetLogger.LogWarning( string.Format( @@ -162,6 +161,39 @@ await GetLatestVersionInternalAsync( $"{packageMetadata.Identity.Id}@{packageMetadata.Identity.Version}", source.Source)); _nugetLogger.LogDebug($"Details: {e}."); + throw new DownloadException(packageMetadata.Identity.Id, packageMetadata.Identity.Version.ToNormalizedString(), new[] { source.Source }, e); + } + finally + { + TryDeleteFile(temporaryFilePath); + } + + async Task VerifyPackageAsync(string packagePath, CancellationToken cancellationToken) + { + if (!_packageVerifier.IsEnabled) + { + return; + } + + ISettings settings = NuGetSettings.LoadDefaultSettings(Directory.GetCurrentDirectory()); + await _packageVerifier.VerifyAsync(packagePath, repository, settings, _nugetLogger, cancellationToken).ConfigureAwait(false); + } + + void TryCopyFile(string sourceFilePath, string destinationFilePath) + { + try + { + _environmentSettings.Host.FileSystem.FileCopy(sourceFilePath, destinationFilePath, overwrite: true); + } + catch (Exception) + { + TryDeleteFile(destinationFilePath); + throw; + } + } + + void TryDeleteFile(string filePath) + { try { _environmentSettings.Host.FileSystem.FileDelete(filePath); @@ -174,7 +206,6 @@ await GetLatestVersionInternalAsync( filePath)); _nugetLogger.LogDebug($"Details: {ex}."); } - throw new DownloadException(packageMetadata.Identity.Id, packageMetadata.Identity.Version.ToNormalizedString(), new[] { source.Source }, e.InnerException); } } @@ -234,12 +265,6 @@ internal IEnumerable LoadNuGetSources(string packageIdentifier, I throw new InvalidNuGetSourceException($"Failed to load NuGet sources configured for the folder {currentDirectory}", ex); } - PackageSourceMapping packageSourceMapping = PackageSourceMapping.GetPackageSourceMapping(settings); - if (packageSourceMapping.IsEnabled && additionalSources.Any(source => !string.IsNullOrWhiteSpace(source))) - { - throw new InvalidNuGetSourceException("Additional sources cannot be used when package source mapping is enabled"); - } - List customSources = new List(); foreach (string source in additionalSources) { @@ -262,6 +287,7 @@ internal IEnumerable LoadNuGetSources(string packageIdentifier, I } IReadOnlyList enabledSources = defaultSources; + PackageSourceMapping packageSourceMapping = PackageSourceMapping.GetPackageSourceMapping(settings); if (packageSourceMapping.IsEnabled) { var mappedSourceNames = new HashSet( @@ -275,11 +301,6 @@ internal IEnumerable LoadNuGetSources(string packageIdentifier, I IEnumerable retrievedSources = customSources.Concat(defaultSources); if (!retrievedSources.Any()) { - if (!enabledSources.Any()) - { - _nugetLogger.LogError(LocalizableStrings.NuGetApiPackageManager_Error_NoSources); - throw new InvalidNuGetSourceException("No NuGet sources are defined or enabled"); - } if (packageSourceMapping.IsEnabled) { throw new PackageNotFoundException( @@ -287,6 +308,8 @@ internal IEnumerable LoadNuGetSources(string packageIdentifier, I enabledSources.Select(source => source.Source), packageSourceMappingMatchNotFound: true); } + _nugetLogger.LogError(LocalizableStrings.NuGetApiPackageManager_Error_NoSources); + throw new InvalidNuGetSourceException("No NuGet sources are defined or enabled"); } return retrievedSources; } @@ -341,7 +364,8 @@ await Task.WhenAll( var accumulativeSearchResults = foundPackagesBySource .Where(result => result.FoundPackages != null) - .SelectMany(result => result.FoundPackages.Select(package => (result.Source, package))); + .SelectMany(result => result.FoundPackages.Select(package => (result.Source, package))) + .Where(result => string.Equals(result.package.Identity.Id, packageIdentifier, StringComparison.OrdinalIgnoreCase)); (PackageSource, NugetPackageMetadata)? latestVersion = accumulativeSearchResults.Aggregate( ((PackageSource, NugetPackageMetadata)?)null, @@ -410,7 +434,8 @@ await Task.WhenAll( continue; } atLeastOneSourceValid = true; - NugetPackageMetadata? matchedVersion = foundPackages.FirstOrDefault(package => package.Identity.Version == packageVersion); + NugetPackageMetadata? matchedVersion = foundPackages.FirstOrDefault(package => + string.Equals(package.Identity.Id, packageIdentifier, StringComparison.OrdinalIgnoreCase) && package.Identity.Version == packageVersion); if (matchedVersion != null) { _nugetLogger.LogDebug($"{packageIdentifier}@{packageVersion} was found in {foundSource.Source}."); diff --git a/src/Microsoft.TemplateEngine.Edge/Microsoft.TemplateEngine.Edge.csproj b/src/Microsoft.TemplateEngine.Edge/Microsoft.TemplateEngine.Edge.csproj index 0653bc7edfc..5e4a6c1b911 100644 --- a/src/Microsoft.TemplateEngine.Edge/Microsoft.TemplateEngine.Edge.csproj +++ b/src/Microsoft.TemplateEngine.Edge/Microsoft.TemplateEngine.Edge.csproj @@ -16,6 +16,7 @@ + diff --git a/test/Microsoft.TemplateEngine.Edge.UnitTests/DotnetNuGetPackageVerifierTests.cs b/test/Microsoft.TemplateEngine.Edge.UnitTests/DotnetNuGetPackageVerifierTests.cs new file mode 100644 index 00000000000..25e8172f393 --- /dev/null +++ b/test/Microsoft.TemplateEngine.Edge.UnitTests/DotnetNuGetPackageVerifierTests.cs @@ -0,0 +1,105 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System.IO.Compression; +using System.Security; +using Microsoft.TemplateEngine.Edge.Installers.NuGet; +using Microsoft.TemplateEngine.TestHelper; +using NuGet.Common; +using NuGet.Configuration; +using NuGet.Protocol; +using NuGet.Protocol.Core.Types; +using Xunit; + +namespace Microsoft.TemplateEngine.Edge.UnitTests +{ + public class DotnetNuGetPackageVerifierTests : IClassFixture, IClassFixture + { + private readonly EnvironmentSettingsHelper _environmentSettingsHelper; + private readonly PackageManager _packageManager; + + public DotnetNuGetPackageVerifierTests(EnvironmentSettingsHelper environmentSettingsHelper, PackageManager packageManager) + { + _environmentSettingsHelper = environmentSettingsHelper; + _packageManager = packageManager; + } + + [Fact] + public async Task VerifyAsync_ValidSignature_Succeeds() + { + string packagePath = await _packageManager.GetNuGetPackage("Newtonsoft.Json", "13.0.3"); + + await VerifyAsync(packagePath); + } + + [Fact] + public async Task VerifyAsync_InvalidSignature_IsRejected() + { + string signedPackage = await _packageManager.GetNuGetPackage("Newtonsoft.Json", "13.0.3"); + string packagePath = Path.Combine(_environmentSettingsHelper.CreateTemporaryFolder(), "Tampered Package.nupkg"); + File.Copy(signedPackage, packagePath); + using (var archive = ZipFile.Open(packagePath, ZipArchiveMode.Update)) + { + using var writer = new StreamWriter(archive.CreateEntry("tampered.txt").Open()); + writer.Write("Content added after signing."); + } + + var exception = await Assert.ThrowsAsync(() => VerifyAsync(packagePath)); + + Assert.Contains("dotnet nuget verify --all", exception.Message); + Assert.Contains("failed with exit code 1", exception.Message); + } + + [Fact] + public async Task VerifyAsync_MissingSignature_AcceptPolicy_Succeeds() + { + string packagePath = CreateUnsignedPackage(); + + await VerifyAsync(packagePath, "accept"); + } + + [Fact] + public async Task VerifyAsync_MissingSignature_RequirePolicy_IsRejected() + { + string packagePath = CreateUnsignedPackage(); + + await Assert.ThrowsAsync(() => VerifyAsync(packagePath, "require")); + } + + private string CreateUnsignedPackage() + { + string packagePath = Path.Combine(_environmentSettingsHelper.CreateTemporaryFolder(), "Unsigned Package.nupkg"); + using (var archive = ZipFile.Open(packagePath, ZipArchiveMode.Create)) + { + using var writer = new StreamWriter(archive.CreateEntry("TestPackage.nuspec").Open()); + writer.Write("TestPackage1.0.0TestTest package"); + } + + return packagePath; + } + + private Task VerifyAsync(string packagePath, string policy = "accept") + { + string directory = _environmentSettingsHelper.CreateTemporaryFolder(); + File.WriteAllText( + Path.Combine(directory, "NuGet.config"), + $""" + + + + + + + + + """); + ISettings settings = global::NuGet.Configuration.Settings.LoadSpecificSettings(directory, "NuGet.config"); + return new DotnetNuGetPackageVerifier().VerifyAsync( + packagePath, + Repository.Factory.GetCoreV3(directory), + settings, + NullLogger.Instance, + CancellationToken.None); + } + } +} diff --git a/test/Microsoft.TemplateEngine.Edge.UnitTests/Mocks/MockNuGetFeed.cs b/test/Microsoft.TemplateEngine.Edge.UnitTests/Mocks/MockNuGetFeed.cs new file mode 100644 index 00000000000..9ba0149fbaa --- /dev/null +++ b/test/Microsoft.TemplateEngine.Edge.UnitTests/Mocks/MockNuGetFeed.cs @@ -0,0 +1,170 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System.Collections.Concurrent; +using System.Net; +using System.Net.Sockets; +using System.Text; +using Newtonsoft.Json; + +namespace Microsoft.TemplateEngine.Edge.UnitTests.Mocks +{ + internal sealed class MockNuGetFeed : IDisposable + { + private readonly TcpListener _listener = new TcpListener(IPAddress.Loopback, 0); + private readonly CancellationTokenSource _cancellation = new CancellationTokenSource(); + private readonly Task _serverTask; + private readonly string _metadataId; + + internal MockNuGetFeed(string metadataId) + { + _metadataId = metadataId; + _listener.Start(); + int port = ((IPEndPoint)_listener.LocalEndpoint).Port; + BaseAddress = $"http://127.0.0.1:{port}/"; + _serverTask = ServeAsync(); + } + + internal string BaseAddress { get; } + + internal ConcurrentQueue Requests { get; } = new ConcurrentQueue(); + + internal byte[] PackageContent { get; } = Encoding.UTF8.GetBytes("downloaded package"); + + public void Dispose() + { + _cancellation.Cancel(); + _listener.Stop(); + try + { + _serverTask.GetAwaiter().GetResult(); + } + finally + { + _cancellation.Dispose(); + } + } + + private async Task ServeAsync() + { + try + { + while (!_cancellation.IsCancellationRequested) + { + using TcpClient client = await _listener.AcceptTcpClientAsync(); + using CancellationTokenRegistration registration = _cancellation.Token.Register(client.Close); + using NetworkStream stream = client.GetStream(); + using StreamReader reader = new StreamReader(stream, Encoding.ASCII, false, 1024, leaveOpen: true); + string? requestLine = await reader.ReadLineAsync(); + if (requestLine is null) + { + continue; + } + + while (!string.IsNullOrEmpty(await reader.ReadLineAsync())) + { + } + + string[] requestParts = requestLine.Split(' '); + if (requestParts.Length != 3 + || requestParts[0] != "GET" + || !Uri.TryCreate(new Uri(BaseAddress), requestParts[1], out Uri? requestUri)) + { + WriteResponse(stream, HttpStatusCode.BadRequest, Encoding.UTF8.GetBytes("Invalid HTTP request."), "text/plain"); + continue; + } + + string path = requestUri.AbsolutePath; + Requests.Enqueue(path); + if (path.StartsWith("/flatcontainer/", StringComparison.Ordinal) + && path.EndsWith(".nupkg", StringComparison.OrdinalIgnoreCase)) + { + WriteResponse(stream, HttpStatusCode.OK, PackageContent, "application/octet-stream"); + } + else if (GetResponse(path) is string response) + { + WriteResponse(stream, HttpStatusCode.OK, Encoding.UTF8.GetBytes(response), "application/json"); + } + else + { + WriteResponse(stream, HttpStatusCode.NotFound, Encoding.UTF8.GetBytes("Unknown feed endpoint."), "text/plain"); + } + } + } + catch (SocketException) when (_cancellation.IsCancellationRequested) + { + } + catch (IOException) when (_cancellation.IsCancellationRequested) + { + } + catch (ObjectDisposedException) when (_cancellation.IsCancellationRequested) + { + } + } + + private static void WriteResponse(NetworkStream stream, HttpStatusCode status, byte[] content, string contentType) + { + byte[] headers = Encoding.ASCII.GetBytes( + $"HTTP/1.1 {(int)status} {status}\r\nContent-Type: {contentType}\r\nContent-Length: {content.Length}\r\nConnection: close\r\n\r\n"); + stream.Write(headers, 0, headers.Length); + stream.Write(content, 0, content.Length); + } + + private string? GetResponse(string path) + { + if (path.StartsWith("/flatcontainer/", StringComparison.Ordinal) + && path.EndsWith("/index.json", StringComparison.Ordinal)) + { + return """{"versions": ["1.0.0"]}"""; + } + string id = JsonConvert.SerializeObject(_metadataId); + if (path.StartsWith("/registration/", StringComparison.Ordinal)) + { + return $$""" + { + "@id": "{{BaseAddress}}registration/templates/index.json", + "count": 1, + "items": [{ + "@id": "{{BaseAddress}}registration/templates/page.json", + "count": 1, + "lower": "1.0.0", + "upper": "1.0.0", + "items": [{ + "@id": "{{BaseAddress}}registration/templates/1.0.0.json", + "catalogEntry": { + "@id": "{{BaseAddress}}catalog/package.json", + "id": {{id}}, + "version": "1.0.0", + "authors": "Test", + "description": "Test package", + "listed": true + }, + "packageContent": "{{BaseAddress}}flatcontainer/templates/1.0.0/templates.1.0.0.nupkg" + }] + }] + } + """; + } + if (path == "/query") + { + return $$""" + {"totalHits": 1, "data": [{"id": "Templates", "version": "1.0.0", "owners": ["Test"]}]} + """; + } + if (path != "/index.json") + { + return null; + } + return $$""" + { + "version": "3.0.0", + "resources": [ + {"@id": "{{BaseAddress}}registration/", "@type": "RegistrationsBaseUrl/3.6.0"}, + {"@id": "{{BaseAddress}}query", "@type": "SearchQueryService/3.4.0"}, + {"@id": "{{BaseAddress}}flatcontainer/", "@type": "PackageBaseAddress/3.0.0"} + ] + } + """; + } + } +} diff --git a/test/Microsoft.TemplateEngine.Edge.UnitTests/Mocks/MockPackageManager.cs b/test/Microsoft.TemplateEngine.Edge.UnitTests/Mocks/MockPackageManager.cs index 1abe6ce7342..a4361a8d7fd 100644 --- a/test/Microsoft.TemplateEngine.Edge.UnitTests/Mocks/MockPackageManager.cs +++ b/test/Microsoft.TemplateEngine.Edge.UnitTests/Mocks/MockPackageManager.cs @@ -13,8 +13,8 @@ namespace Microsoft.TemplateEngine.Edge.UnitTests.Mocks internal class MockPackageManager : IDownloader, IUpdateChecker { internal const string DefaultFeed = "test_feed"; - internal const string PackageSourceMappingMatchNotFound = "PackageSourceMappingMatchNotFound"; private readonly NuGetPackageInfo? _downloadResult; + internal const string PackageSourceMappingMatchNotFound = "PackageSourceMappingMatchNotFound"; private readonly PackageManager? _packageManager; private readonly string? _packageToPack; diff --git a/test/Microsoft.TemplateEngine.Edge.UnitTests/NuGetApiPackageManagerTests.cs b/test/Microsoft.TemplateEngine.Edge.UnitTests/NuGetApiPackageManagerTests.cs index a59027d29f0..e74ff5492fe 100644 --- a/test/Microsoft.TemplateEngine.Edge.UnitTests/NuGetApiPackageManagerTests.cs +++ b/test/Microsoft.TemplateEngine.Edge.UnitTests/NuGetApiPackageManagerTests.cs @@ -4,9 +4,14 @@ using FluentAssertions; using Microsoft.TemplateEngine.Abstractions; using Microsoft.TemplateEngine.Edge.Installers.NuGet; +using Microsoft.TemplateEngine.Edge.UnitTests.Mocks; using Microsoft.TemplateEngine.TestHelper; +using NuGet.Common; using NuGet.Configuration; +using NuGet.Protocol.Core.Types; +using NuGet.Versioning; using Xunit; +using NuGetSettings = NuGet.Configuration.Settings; namespace Microsoft.TemplateEngine.Edge.UnitTests { @@ -20,6 +25,49 @@ public NuGetApiPackageManagerTests(EnvironmentSettingsHelper environmentSettings _environmentSettingsHelper = environmentSettingsHelper; } + [Theory] + [InlineData("1.0.0")] + [InlineData(null)] + [InlineData("*")] + [InlineData("1.*")] + [InlineData("*-*")] + public async Task DownloadPackage_MismatchedMetadataId_ThrowsPackageNotFound(string? version) + { + const string identifier = "Requested.Package"; + using var feed = new MockNuGetFeed("Other.Package"); + string source = feed.BaseAddress + "index.json"; + string installPath = _environmentSettingsHelper.CreateTemporaryFolder(); + var settings = _environmentSettingsHelper.CreateEnvironment(virtualize: true); + var manager = new NuGetApiPackageManager(settings); + + var exception = await Assert.ThrowsAsync(() => manager.DownloadPackageAsync( + installPath, identifier, version, additionalSources: new[] { source }, force: true)); + + exception.PackageIdentifier.Should().Be(identifier); + NuGetVersion.TryParse(version, out NuGetVersion? expectedVersion); + Assert.Equal(expectedVersion, exception.PackageVersion); + exception.SourcesList.Should().Equal(source); + Directory.GetFiles(installPath).Should().BeEmpty(); + feed.Requests.Should().NotContain(path => path.StartsWith("/flatcontainer/", StringComparison.Ordinal)); + } + + [Fact] + public async Task GetLatestVersion_MismatchedMetadataId_ThrowsPackageNotFound() + { + const string identifier = "Requested.Package"; + using var feed = new MockNuGetFeed("Other.Package"); + string source = feed.BaseAddress + "index.json"; + var settings = _environmentSettingsHelper.CreateEnvironment(virtualize: true); + var manager = new NuGetApiPackageManager(settings); + + var exception = await Assert.ThrowsAsync(() => manager.GetLatestVersionAsync( + identifier, additionalSource: source)); + + exception.PackageIdentifier.Should().Be(identifier); + exception.PackageVersion.Should().BeNull(); + exception.SourcesList.Should().Equal(source); + } + [Fact] public async Task DownloadPackage_Latest() { @@ -41,6 +89,29 @@ public async Task DownloadPackage_Latest() result.Reserved.Should().BeFalse(); result.PackageVersion.Should().NotBeNullOrEmpty(); result.NuGetSource.Should().NotBeNullOrEmpty(); + Directory.EnumerateFiles(installPath, "*.tmp").Should().BeEmpty(); + } + + [Fact] + public async Task DownloadPackage_PackageValidationFailure_RemovesTemporaryPackage() + { + string installPath = _environmentSettingsHelper.CreateTemporaryFolder(); + IEngineEnvironmentSettings engineEnvironmentSettings = _environmentSettingsHelper.CreateEnvironment(virtualize: true); + + NuGetApiPackageManager packageManager = new NuGetApiPackageManager( + engineEnvironmentSettings, + new FailingNuGetPackageVerifier()); + + var exception = await Assert.ThrowsAsync(() => packageManager.DownloadPackageAsync( + installPath, + "Microsoft.DotNet.Common.ProjectTemplates.5.0", + "5.0.0", + additionalSources: new[] { "https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json" })); + + var validationException = Assert.IsType(exception.InnerException); + validationException.Message.Should().Be("Package validation failed."); + + Directory.EnumerateFiles(installPath).Should().BeEmpty(); } [Fact] @@ -313,36 +384,36 @@ public void LoadNuGetSources_PackageSourceMappingDisabled_ReturnsAllSources() sources.Select(source => source.Name).Should().BeEquivalentTo("template-feed", "other-feed"); } - [Theory] - [InlineData("https://additional.test/v3/index.json")] - [InlineData("https://templates.test/v3/index.json")] - public void LoadNuGetSources_PackageSourceMappingEnabledAndAdditionalSourceSpecified_Throws(string additionalSource) + [Fact] + public void LoadNuGetSources_AdditionalSourceSpecified_AllowsAdditionalSourceAndFiltersConfiguredSources() { IEngineEnvironmentSettings engineEnvironmentSettings = _environmentSettingsHelper.CreateEnvironment(virtualize: true); NuGetApiPackageManager packageManager = new NuGetApiPackageManager(engineEnvironmentSettings); ISettings settings = CreateSettingsWithPackageSourceMapping(); - Action action = () => packageManager.LoadNuGetSources( + IEnumerable sources = packageManager.LoadNuGetSources( "Contoso.Templates", - new[] { additionalSource }, + new[] { "https://additional.test/v3/index.json" }, settings); - action.Should().Throw(); + sources.Should().ContainSingle() + .Which.Source.Should().Be("https://additional.test/v3/index.json"); } [Fact] - public void LoadNuGetSources_PackageSourceMappingEnabledAndNoSourcesEnabled_ThrowsInvalidNuGetSourceException() + public void LoadNuGetSources_ConfiguredAdditionalSource_DoesNotBypassPackageSourceMapping() { IEngineEnvironmentSettings engineEnvironmentSettings = _environmentSettingsHelper.CreateEnvironment(virtualize: true); NuGetApiPackageManager packageManager = new NuGetApiPackageManager(engineEnvironmentSettings); - ISettings settings = CreateSettingsWithPackageSourceMapping( - disableAllSources: true); + ISettings settings = CreateSettingsWithPackageSourceMapping(); - Action action = () => packageManager.LoadNuGetSources( - "Microsoft.TemplateEngine.TestTemplates", - Array.Empty(), + IEnumerable sources = packageManager.LoadNuGetSources( + "Exact.Templates", + new[] { "https://templates.test/v3/index.json" }, settings); - action.Should().Throw(); + + sources.Should().ContainSingle() + .Which.Name.Should().Be("other-feed"); } [Fact] @@ -408,18 +479,10 @@ public void RemoveInsecurePackages_Mixed() private ISettings CreateSettingsWithPackageSourceMapping( bool disableTemplateSource = false, - bool disableAllSources = false, bool includePackageSourceMapping = true) { string settingsDirectory = _environmentSettingsHelper.CreateTemporaryFolder(); - string disabledSources = disableAllSources - ? """ - - - - - """ - : disableTemplateSource + string disabledSources = disableTemplateSource ? """ @@ -455,7 +518,17 @@ private ISettings CreateSettingsWithPackageSourceMapping( {packageSourceMapping} """); - return global::NuGet.Configuration.Settings.LoadSpecificSettings(settingsDirectory, "NuGet.config"); + return NuGetSettings.LoadSpecificSettings(settingsDirectory, "NuGet.config"); + } + + private sealed class FailingNuGetPackageVerifier : INuGetPackageVerifier + { + public bool IsEnabled => true; + + public Task VerifyAsync(string packagePath, SourceRepository repository, ISettings settings, ILogger logger, CancellationToken cancellationToken) + { + throw new InvalidDataException("Package validation failed."); + } } } } diff --git a/test/Microsoft.TemplateEngine.Mocks/MockFileSystem.cs b/test/Microsoft.TemplateEngine.Mocks/MockFileSystem.cs index 0d9c1a358f1..21d048c3471 100644 --- a/test/Microsoft.TemplateEngine.Mocks/MockFileSystem.cs +++ b/test/Microsoft.TemplateEngine.Mocks/MockFileSystem.cs @@ -157,7 +157,7 @@ public void FileCopy(string sourcePath, string targetPath, bool overwrite) throw new Exception($"File {sourcePath} doesn't exist"); } - _files[targetPath] = new FileSystemFile(_files[targetPath]); + _files[targetPath] = new FileSystemFile(_files[sourcePath]); } public IEnumerable EnumerateFileSystemEntries(string directoryName, string pattern, SearchOption searchOption) diff --git a/tools/Shared/Microsoft.TemplateEngine.CommandUtils/ArgumentEscaper.cs b/tools/Shared/Microsoft.TemplateEngine.CommandUtils/ArgumentEscaper.cs index eb0d24b4045..5a52c8b98e9 100644 --- a/tools/Shared/Microsoft.TemplateEngine.CommandUtils/ArgumentEscaper.cs +++ b/tools/Shared/Microsoft.TemplateEngine.CommandUtils/ArgumentEscaper.cs @@ -106,13 +106,22 @@ internal static bool ShouldSurroundWithQuotes(string argument) internal static bool IsSurroundedWithQuotes(string argument) { +#if NET return argument.StartsWith('"') && argument.EndsWith('"'); +#else + return argument.StartsWith("\"", StringComparison.Ordinal) && + argument.EndsWith("\"", StringComparison.Ordinal); +#endif } internal static bool ArgumentContainsWhitespace(string argument) { +#if NET return argument.Contains(' ') || argument.Contains('\t') || argument.Contains('\n'); +#else + return argument.Contains(" ") || argument.Contains("\t") || argument.Contains("\n"); +#endif } ///