UTF-8 vs ANSI – Errors with < : & and stacks of other characters

I just had an interesting experience trying to troubleshoot some problems with a PowerShell script that would work just fine when copied and pasted from Notepad++ into the PowerShell window, but would error out when run from the .PS1 file. It seems that nested variables would “get weird” when they were run from a UTF-8 file in PowerShell.

What was I doing with the code that was failing?

  • I was building out a HTML string in a variable and then using the string as the body of a HTML email which I was sending using the Graph API.

What tests had I run to try and eliminate large components?

  • The same HTML string worked fine when I sent it as an email using PowerShell’s Send-MailMessage (so, something to do with the way JSON was encoding the string maybe?),
  • The HTML string was emailed using the Graph API just fine when copied and pasted into the Powershell console (the same code works in the console but fails in the ps1 file),
  • The errors were all related to the mail function, and
  • I went through and tried to replace the “” by escaping a single set of double quotes with \”, then I replaced “” with ‘ to see if it made a difference

Could I narrow the scope down any more?

  • Remove the Function that sent the email, and the .ps1 script file ran fine
  • The HTML string being produced for the email body was valid HTML
  • The errors were being thrown at compile time, not during the execution of the script

Specifically I was getting these problems (in bold):

At E:\OneDrive\PowerShell\SyncSelfElevationRightsWithADGroupMembership_AzureMailServer.ps1:484 char:267
+ ... n'>Exception! User account permanently added to Role and 
 is ...
+                                                                 ~
The '<' operator is reserved for future use.
 
At E:\OneDrive\PowerShell\SyncSelfElevationRightsWithADGroupMembership_AzureMailServer.ps1:533 char:16
+             "isElevated": false,
+                         ~
Unexpected token ':' in expression or statement.
 
At E:\OneDrive\PowerShell\SyncSelfElevationRightsWithADGroupMembership_AzureMailServer.ps1:555 char:43
+ ...    $EmailString = "Unexpected token 'bgcolor='#FFFFFF'' in expression or statement.
 
At E:\OneDrive\PowerShell\SyncSelfElevationRightsWithADGroupMembership_AzureMailServer.ps1:556 char:117
+ ... br />Your friendly Office365 Platform Support Team
"
+                                                                         ~
The string is missing the terminator: ".
 
At E:\OneDrive\PowerShell\SyncSelfElevationRightsWithADGroupMembership_AzureMailServer.ps1:412 char:9
+     try {
+         ~
Missing closing '}' in statement block or type definition.

At E:\OneDrive\PowerShell\SyncSelfElevationRightsWithADGroupMembership_AzureMailServer.ps1:562 char:1
+
The Try statement is missing its Catch or Finally block.
 
At E:\OneDrive\PowerShell\SyncSelfElevationRightsWithADGroupMembership_AzureMailServer.ps1:363 char:33
+ function Send-NotificationEmail {
+                                 ~
Missing closing '}' in statement block or type definition.

None of these errors were valid! The syntax and bracketing etc. were fine (intellisense told me so, and it would have errored in the Powershell window when I ran it in there).

Finally, after much googling and swearing, I came across this Technet Forum discussion which mentioned the powershell file stopped working when it was saved in something other than ANSI… so I gave it a shot.

The first file executed in this screenshot is saved in UTF-8 file format. The second one is exactly the same file, but saved in ANSI format (even though it failed the AzureAD Module test, it compiled and ran without erroring).

image001

UTF-8 vs ANSI

My current working theory is the original file had some code copied from another Robocopy file (probably in UTF-8 format) which may have had an extended hyphen character or an unusual whitespace character in it. As soon as I saved it as AzureMailServer2.ps1 in ANSI format, it threw away all of the characters that were not ANSI compliant and worked just fine.

Just have to chalk that one up to experience I guess… 😊

About Brad Saide

I'm a SharePoint consultant. I'm also slowly going bald, seem to have a permanent spare tyre around my waist and enjoy socialising with friends over a beer or 10. The last 2 may possibly be related. Started working with SharePoint when the first version was in limited beta release (participated in the Technology Adoption Program while at Woolworths) and have been committed to the adoption of the technology as a business enabler ever since.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a comment