The New Microsoft Partner Center (Le Sigh)

Microsoft's latest regurgitation of their Partner Portal is the Microsoft Partner Center.  This is a fair bit different to previous regurgitations as it now requires an Azure AD account to authenticate against.  For newer Partners this won't really be an issue as they will have a single Azure AD tenant that is where their MPN Azure Credits and also their IUR Office 365 rights are assigned.  Most likely.

But for those of us who have been around since before Azure/BPOS/Office 365 existed, that's likely a very different story.  :(

Way back in prehistoric times, many Microsoft Partners had a Live ID (later known as a Personal Account) that they signed into the Partner Portal with.  Then Microsoft came up with Azure and created a tenant based on that Live ID and failed to mention this to their Partner Channel.  So, for instance, if your original Live ID was FSM@rocks.com then a new tenant named "fsmrocks" was created and it was here that the Azure Credits were assigned.

Since many of us longer-term Partners signed up for BPOS (what Office 365 was originally known as) before taking a serious look at Azure, many of us followed the procedure as we'd do for our clients - create a tenant (such as fsm.onmicrosoft.com), set up a custom domain, assign the License Key here (in this case the IUR key) and away you go.  This then created an Azure AD tenant and the world was good.

Or so we thought...

Now there was an Azure AD tenant for your MPN and Azure credits (fsmrocks) and another (fsm.onmicrosoft.com) that had all your real users and the custom domain you used regularly (fsm.com).  And unless you really needed to have Azure guests working with Azure AD COnnect back to your on-premises domain, that was also a non-issue.

Now Partner Center is introduced.  It needs to use an Azure AD account.  But which one?  Which tenant?

The more switched on members of the long-term MS Partner crowd here will see this will be an issue.  And another Partner issue is just what we don't need.  Yet another Partner issue...

So, here we go down the complicated, time consuming and unnecessarily frustrating rabid hole.  (Yes, I know - it really is rabid, though...)

You will need to "merge" the tenants.  If you don't have anything you're not willing to rebuild (or export and import) in Azure on the MPN account, then that's going to be a lot easier.  So you have a source tenant (MPN tenant - in this case fsmrocks) and a destination tenant (Partner Center tenant - in this case fsm.onmicrosoft.com).  We will need to migrate your older MPN account to the new Partner Center (relatively easy) and then create an Azure Support Ticket and have the MPN subscription moved to this new Partner Center tenant (unnecessarily frustrating and difficult).

So, either calling Microsoft Support (hhmmm...) or using their online support you will get the existing MPN account converted to a new Partner Center account.  The primary login for this Partner Account needs to be a user with Admin rights in the destination tenant, so we'll use FSMisLord@fsm.onmicrosoft.com as an example.  This account needs an email box assigned for the duration of this exercise, which can then be removed.  Once this has finished, you'll be able to log into both https://Portal.Office.com and also https://My.VisualStudio.com with this account, however the My.VisualStudio site won't have your MPN details assigned yet.

Now the fun really starts.

Signing into the source Azure Portal, see if you can go into Subscriptions and initiate a move of the MPN subscription to the destination tenant (

https://docs.microsoft.com/en-us/azure/billing/billing-subscription-transfer

).  I'm betting it fails.  So kick off a support ticket and ask to have your MPN subscription moved to the new Partner Centre account.  This is where things really can go South.

Quite probably, you won't have a Partner Center account that Azure can see.  I don't know if the Azure team yet know the Partner Center is using Azure AD credentials.  I'm sure in a few years or so they will work it out.  Thre Azure team should contact you within 24 hours and will likely have no idea what you are trying to do, why, and how to fix it effectively.

So after prompting by the Azure Support call center contractor, from the source tenant email address (the old Live ID) you'll eventually need to send them a lot of information - the MPN Subscription ID you wish to move, the physical address of the old tenant, the last 4 digits of the CC used on the old tenant tenant, the email address of the administrator of the new tenant, some screenshots of you logged into the new Partner Center so they can see you can log in, some screenshots of the My.VisualStudio.com login with the destination tenant admin account showing there is no MPN stuff assigned yet, and maybe something else, depending on which call center contractor you get hold of.

Then from the email account of the Partner Center admin (in this case, FSMisLord@fsm.onmicrosoft.com) you need to send the same details as you sent for the source tenant, but with destination tenant information (CC, email of the source tenant, MPN subscription ID, fully worked proof of Legendre's Conjecture, and possibly some other random information about your neighbors across the road).

Eventually, a few dozen emails and days later (hopefully a lot less for you since I have already detailed a lot of information above to help this procedure out) it should be sorted.

If you need to know where to send a bottle of bourbon, contact me for the details.  :)

Regards,

The Outspoken Wookie

CIAOPS Microsoft Cloud Course

As an IT provider to SMB clients, do you look up and see this wibbly wobbly cloudy woudy thing and wonder what it actually is?  Do you look at Microsoft Office 365 Business and Microsoft 365 Business and wonder what the difference is, except one includes Windows 10 which your clients already own?  Are you looking at the Google offerings and wondering what Microsoft could possibly do better?

Or maybe you are rather hoopy and wish to become quite a frood regarding Microsoft's Cloud offerings?

Well, either way, Robert Crane (yes, THAT Robert Crane) from CIAOPS (no, not the real CIA, they already know all about you) who is a really hoopy frood can help you learn more about what Microsoft has to offer the SMBs using their Cloud services.

So, if you're in or around Sydney on 19th July and are wondering which cafe to sip latte at for the day, how about heading over to Cliftons on Margaret Street instead for a great day of learning?  If you want more information on this course, have a read of this page, and if you really love the idea of Robert being able to eat from his efforts and receive a lot more information from him regularly (not necessarily just about what he eats), then have a look here.

Regards,

The Outspoken Wookie

Bootable Windows Server 2016 UEFI ISO

This is an update of my http://hiltont.blogspot.com.au/2013/09/bootable-windows-server-2012-iso.html post.

Back in the Windows F^HVista/7 and Server 2008/2008R2 days, it was possible to install an operating system instance without needing to input a key during the installation process.  You could use various methods to make a bootable OS Install USB key then simply remove the \sources\ei.cfg file and you'd be able to choose between all the versions of the OS inside the boot.wim file - this was very handy!

Microsoft has made a change since Windows Server 2012/Windows 8 whereby they don't allow you to install Windows without the use of a Product Key.  There is no ei.cfg file in the \sources folder and you are forced to input a Product Key during installation.

Well, there's a simple way to work around this to not only create a bootable USB Key that doesn't need a Product Key during install (and lets you choose among the multiple OS variants in the boot.wim file), but also to re-create a bootable ISO image so that you can then mount this as an installation media source in Hyper-V!

The steps to achieve these goals are:

• First, you'll need to download a Windows Server 2012, 2012R2 or 2016 ISO image or use the VL ISO image you already have there.  Then you'll need to either extract this image to a folder, or better still, just run the (portable) Rufus application to make yourself a bootable USB key that allows for quite quick OS installs.
  
• Now, you'll need to copy the following into a new plain text file called ei.cfg in the \sources folder of the USB key:
  [Channel]
  Retail
  [VL]
  1
• If you only need the bootable USB key, you can stop here, however if you now wish to recreate the ISO image file so that you can then use this modified image in a Hyper-V environment...
• Download and install the latest Windows Assessment and Deployment Kit (ADK) (the latest at time of writing this post was for Windows 10 Build 1607).  The installer itself is around 1.7MB and the only component you need to install for this process is Deployment Tools, which is around 68MB
• Now, to create an ISO image that will work with BIOS-based systems, assuming the USB Key we just made is available at U: and you're creating the ISO image on D:
  
  • Run the Deployment and Imaging Tools Environment (Command Prompt) as Administrator, change to the "amd64\oscdimg" folder and then type the following:
  • oscdimg.exe –l<disc_label> -m -u2 –b<path_of_bootable_img_file> <path_of_installation_source> <path_where_ISO_is_to_be_created>
  • Please see this link for an explanation of the oscdimg.exe command-line options
  • Example:
    oscdimg.exe -lSVR2016BIOS -m -u2 -b"U:\boot\etfsboot.com" "U:\" "D:\Svr2016BIOS.iso"
• If you wish to create an ISO image that will work on both BIOS-based and UEFI-based systems such as a Hyper-V 2012 R2 and 2016 Gen 2 guest instance, assuming the USB Key we just made is available at U: and you're creating the ISO image on D: (this uses a modified version of the instructions found here):
  
  • Run the Deployment and Imaging Tools Environment (Command Prompt) as Administrator, change to the "amd64\oscdimg" folder and then type the following:
  • oscdimg -l2016UEFI -m -o -u2 -udfver102 -bootdata:2#p0,e,b"U:\boot\etfsboot.com"#pEF,e,b"U:\efi\microsoft\boot\efisys.bin" "U:\" "D:\2016-ei-UEFI.iso"
  • Please see the original post (here) for more information on the command-line options used.
• Now, you can burn this ISO image to a DVD or simply use it as the source for a Hyper-V guest and because it contains both the BIOS and UEFI boot data, it can be used to install either a Gen 1 (BIOS) or Gen 2 (UEFI) Hyper-V Guest. :)

Enjoy!


Regards,

The Outspoken Wookie

Microsoft Action Pack Subscription Promo Codes

Does anyone need to renew their MAPS (Microsoft Action Pack Subscription)? If so, when you go to pay, use the Promo Code "WELCOME2ACTIONPACK" for a 50% discount and then once you have entered this one, enter "GROWYOURBIZWITHMAPS" for a further 50% discount - that brought it from $490 ex tax down to $134.20 inc tax! :)

Regards,

The Outspoken Wookie

Mar 2016 Update to .NET Redistributables

I have just updated my http://hiltont.blogspot.com.au/2008/01/net-redistributables.html post with links to the latest redistributables for dotNET 4.6 and 4.6.1.  :)

Regards,

The Outspoken Wookie

Accessing O365 Shared Mailboxes With Android

Microsoft's official position is that you cannot access Shared Mailboxes in an Office 365 tenancy from an Android device.  Here's how to do it by configuring a new IMAP account with the following settings:

Incoming Settings
1. Email Address = SharedMailbox@Domain.com
2. Username = O365Account@Domain.com\SharedMailbox
3. Password = Password
4. IMAP Server = outlook.office365.com
5. Security Type = SSL
6. Port = 993
7. IMAP path prefix = Optional

Outgoing Settings
1. SMTP Server = smtp.office365.com
2. Security Type = TLS
3. Port = 587
4. Require Sign In = Yes
5. Username = O365Account@Domain.com
6. Password = Password

You should be able to have this work also on iOS and Android (and maybe even Mac OSX) devices by setting an account up in a similar fashion.

Regards,

The Outspoken Wookie

Windows 7 Reaches Its First EOL Step

Windows 7, the operating system that Microsoft simply can't kill off, has reached the first major step in its End Of Life cycle - the end of mainstream support was yesterday (13 Jan, 2015).  As can be seen on Microsoft's Product Support Lifecycle page for Windows 7, this date has now been reached and we're now in the Extended Support phase.  This is just like when Voyager 1 passed out of our Solar System in August 2012 and into interstellar space.

What this means is that the product will now only receive security updates - no non-security updates will be made available for Windows 7 based products (unless you have a Premier Support Agreement for Windows 7 Enterprise, which I can guarantee no small business has).  This is not a nail in the coffin of Windows 7, but it does mean that the product is entering its final years of functionality.  Paid support is still available for Windows 7 on a per-incident basis.

On the 14th of January, 2020 we'll have another milestone - the last day of extended (security update) support.  That will be the last nail in the coffin of Windows 7 - this is what happened to Windows XP back on 8th April, 2014.

So, right now there's nothing to worry about if you're still running Windows 7 on your computers - currently computer manufacturers are still allowed to supply computers with Windows 7 pre-installed if the manufacturer still has old stock of Windows 7 licenses.  But with Windows 8 and also Windows 8.1 already out and Windows 10 on the horizon, it may be time to think about the steps needed to move up to the current/next version of Windows.

Regards,

The Outspoken Wookie

Outright theft and plagiarism

It was brought to my attention yesterday that the author (sic) Pomiraja from the http://www.AskMeBoy.com site was plagiarising my blog posts.  I'd like to thank the anonymous person who brought this to my attention.

AskMeBoy have so far, through Pomiraja's plagiarism, republished 26 of my blog posts (as of right now) as Pomiraja's own work without giving any credit to me.  To see exactly how stupid this plagiarist is, they have even left my signature on all of the posts they have stolen!

If you would like to see a list of the plagiarised posts, have a look at https://askmeboy.com/?s=outspoken+wookie or the screen captures I took below - as I said, there are 26 at this point in time.  I don't know if this will encourage the owners of the website to axe Pomiraja for this plagiarism, axe the stolen and mis-credited posts, or have the site publish a written apology.  Their contact page is broken so I could not contact them directly about this, forcing me to make this public complaint about their plagiarism.

Regards,

The Outspoken Wookie

BEASTly POODLEs

There have been a number of vulnerabilities detected in various security protocols over the past year or two including BEAST Attack, Heartbleed Bug and POODLE Attack.  At least 2/3 of these have names that give some indication of their severity and the remaining 1/3 leaves you with a rather interesting visual image.  But be ye not distracted by the names - they are all things that need to be addressed in various ways.

Information about the Browser Exploit Against SSL/TLS (BEAST) Attack was released in September 2011 and involved attacking the lack of security in particular implementations of TLS 1.0 traffic.  This vulnerability has been pretty much mitigated today (Dec, 2014), however there are still some older, non-updated systems out there that are vulnerable to this attack.  The table below lists the earliest version of the products that have mitigated the BEAST Attack (and yes, Apple took an inordinately long time to patch for this vulnerability):

Apple iOS	iOS 7.0
Apple OS-X	OS-X 10.9 (Mavericks)
Google Chrome	Version 16
Microsoft Windows	MS12-006 on Windows 7/Server 2008 R2 and older
Mozilla Firefox	Version 10

Following on from the BEAST Attack were the CRIME and BREACH attacks which, too, have been mitigated in current browsers and are a low-grade threat at worst these days.

The Heartbleed bug, publicly announced in April, 2014, affected anything running OpenSSL.  The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software. This compromises the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of the users and the actual content. This allows attackers to eavesdrop on communications, steal data directly from the services and users and to impersonate services and users.  Yup, it is pretty nasty but this, too, is pretty much completely mitigated by the various vendors using this code.

For an easy to understand explanation of the Heartbleed bug, have a read of this XKCD comic.  To see if your website is affected by the Heartbleed bug, have a look at https://lastpass.com/heartbleed/ (Heartbleed bug only) and https://www.ssllabs.com/ssltest/index.html (Heartbleed and more).  Any site that you go to that uses the "https" protocol can be checked to ensure it is running a version of OpenSSL that is not vulnerable to this attack.  If the site *still* has not been updated, I'd suggest speaking with the vendor, outing them in social media and removing your account and changing any passwords and/or information that was stored in that site.

And now we come to what at first glance may be the fluffiest of all these vulnerabilities - the POODLE Attack.  Basically, there's the ability in browsers to request a lower level of security from the server if the browser doesn't support the version the server prefers.  This is called a security renegotation.  The POODLE Attack uses a recently discovered flaw in the now obsoleted and in the process of fast becoming deprecated SSL 3.0 protocol mixed with a renegotiation attack (forcing the server to drop from TLS 1.x to SSL 3.0).  The simple fix is to disable SSL 3.0 on all your web servers, however there are still some applications that use SSL 3.0 (again, speak with the vendor, expose in social media and seriously question your continued trusting of a vendor using 18 year old technology that's been superseded 3 times).

To read more on the POODLE Attack and how to ensure you're doing everything you can to protect against it, have a read of https://scotthelme.co.uk/sslv3-goes-to-the-dogs-poodle-kills-off-protocol/ and then go to https://www.poodlescan.com/ and https://www.ssllabs.com/ssltest/index.html to confirm your server mitigations have been invoked.  There's also a funky little tool from Nartac Software called IISCrypto that can help you properly configure your Windows IIS to mitigate against POODLE and other vulnerabilities.

The table below lists the earliest version of the products that have mitigated the POODLE Attack:

Apple iOS	iOS 8.1
Apple OS-X	OS-X Security Update 2014-005 (Mavericks & Mountain Lion)
Google Android	Chrome - still waiting
Google Android	Samsung Browser - still waiting
Google Chrome	Version 39
Microsoft Windows	Temporary Fix it released, also shows Group Policy fix
Mozilla Firefox	Version 34

If you want to see if your client (browser) is susceptible to the POODLE Attack, go to https://www.poodletest.com/.  If your browser is vulnerable, don't trust it to keep your data secure.


Regards,

The Outspoken Wookie

Azure Needs To Be Introduced To 2012

I'd *REALLY* like to know how, if Microsoft is pumping so much time and energy into Azure, it cannot handle the VHDX format, let alone Generation 2 Virtual Machines from Hyper-V 2012?  I mean, we're now in 2014 and have the Windows Server 10 Technical Preview available to us and Microsoft still can't handle .VHDX files in their Azure virtual machines!

This is slack.  Really slack.  It means that even though we who use on-premises Hyper-V Servers can use the current Microsoft technologies for all of our current-release guests, we cannot do it using their Azure platform.  It means that we cannot even upload our virtual machines nor use SCVMM and replicate our Gen 2 VMs into Azure.

Come on Microsoft - bring the Azure infrastructure up to your current generation before you get severely left behind yourself! :(

Regards,

The Outspoken Wookie

Microsoft's Broken Store Upgrade to 8.1

After spending well over 8 hours trying to work through this issue, including a reasonable amount of Google searching, it seems to me that Microsoft has royally fucked up the Microsoft Store Windows 8.1 Upgrade for Windows 8.0 users.  And now, I'll explain it in a little more depth...

For anyone who needs/wants to reinstall their Windows 8 (RTM), then activate with their license key, then install the updates needed (KB2871389 and KB2917499) to get the Windows 8.1 upgrade from the Microsoft Store and upgrade to Windows 8.1, you're in for a bit of a shock if you need to run applications that need the .NET Framework 3.5 (or 3.0 or 2.0) Feature that's normally installable via either the Control Panel or a particular invocation of the DISM command.

If you try and install via Windows Update, you'll be presented with the following error message:

Windows couldn't connect to the Internet to download necessary files. Error code: 0x800F0906 (which, of course, is bullshit if you've got a functional Internet connection - this is an erroneous error message)

If instead you try the DISM way, you'll see either the above error code or the following:

•0x800F081F: The changes could not be completed.

The reason for this, it seems, is because downloading the Windows 8.1 Upgrade from the Microsoft Store gives you the latest slipstreamed install of Windows 8.1 with almost all current Updates and HotFixes. Normally that would be great as it means that you don't need to bother performing a boatload of updates after installing the Windows 8.1 Upgrade, except that it seems that Microsoft has chosen to include in the slipstreamed Upgrade, HotFixes KB2966826 and KB2966828 (see: http://answers.microsoft.com/en-us/windows/forum/windows8_1-winapps/cant-enable-or-install-net-framework-35-in-windows/2f221b25-8d17-421f-8c37-1fed61649b4f) which are .NET 3.5 updates, meaning that until you uninstall these (which you cannot do if you've installed the 8.1 Upgrade from the Microsoft Store) you are what's known in the trade as SOL if you want to install .NET 3.5 (including 3.0 and 2.0) in a newly installed Win 8.0/8.1 from the Store.

As mentioned in that answers.microsoft.com thread, if these have been installed as part of the regular Microsoft Update regime before .NET 3.5 is even installed, it is a simple enough task to uninstall them, install .NET 3.5 and then reinstall these updates.  This is not the case with the slipstreamed Windows 8.1 Upgrade from the Microsoft Store as these updates seem to have been slipstreamed into the installer, meaning that they don't show as updates that can be uninstalled, therefore they can't be uninstalled, therefore you CANNOT install .NET 3.5 on this system - that leaves a lot of software that won't install nor run properly.

Out of interest, this thread also details issues with these updates - so it isn't just on Microsoft's own site that this issue has been discussed.

I have found - as expected - that even if you install .Net 3.5 in Windows 8.0 before performing the Microsoft Store upgrade to Windows 8.1, Windows 8.1 is installed with .Net 3.5 uninstalled.  As I said, I expected this behaviour and would have been rather surprised if this had actually worked.

It seems to me that the easiest fix for this is to re-release both of these HotFixes with better pre-requisite detection and also rebuild the Microsoft Store upgrade image so that these two HotFixes are not included, or alternatively, fix the Windows Update site so that if a Windows 8.1 system with these updates already installed, but .Net 3.5 not yet installed, goes out looking for the .Net 3.5 installation files, it provides a working set of files that will install the .Net 3.5 Feature.


Until then, good luck if you need to get the .Net 3.5 Feature installed on a Windows 8.1 system recently upgraded from Windows 8.0 through the Microsoft Store.

Regards,

The Outspoken Wookie

Companyweb on SBS 2003 stopping

We've all had those sites where, for some reason, Companyweb just stops, needs restarting then runs fine for a while and then stops again.  Gotta love it...

Well, I have a site we're in the process of migrating from SBS 2003 to Essentials 2012 R2 + Office 365 where this started happening a few days ago, so I decided to sort out a scheduled script to take care of this for me.

@Echo Off
if exist default.aspx del default.aspx

"C:\Program Files\GnuWin32\bin\wget.exe" --user=administrator --password="Seriously, you think I'd post that on my blog?" --timeout=30 --tries=1 http://CompanyWeb/default.aspx

find /C "SharePoint Team Web Site" default.aspx

IF ERRORLEVEL 1 cscript C:\WINDOWS\system32\iisweb.vbs /start companyweb

I'm running this script every 5 minutes which places bugger all extra load on the server, but means that it isn't long after the Companyweb site stops that it is restarted.

Oh, and run it as "nt authority\system" so it doesn't pop up a command prompt window on the user's desktop (ie, the administrator's desktop).

Regards,

The Outspoken Wookie

APNs

OK, so I'm sick and tired of visiting various sites to find the APNs for the carious 3G/4G providers our clients use at work and at home, so here's my own list...

Provider	3G/4G	APN	Account type	Comments	URL
Aldi Mobile	3G	mdata.net.au	Personal	No business use - see 1.4.1.d here.	Aldi Mobile
Amaysim	3G	internet			Amaysim
ApexN	3G			Optus based plans	ApexN
Bigpond	3G/4G	telstra.bigpond		Username and password required
BoostMobile	3G	telstra.internet		Mainly personal use - see 1.1 here.	BoostMobile
Exetel	3G	exetel1	Premium	Older Optus-based plans
Exetel	3G	connect	Standard	Newer Optus-based plans
Exetel	4G	yesinternet		New Optus 4G plans
Internode	3G	internode
Optus	3G	internet		Best option	Optus
Optus	3G	yesinternet		Use if "internet" doesn't work	Optus
Optus	4G	yesinternet			Optus
Pacnet	3G	internet	Static IP	Optus infrastructure	Pacnet
Pacnet	3G	pacnet	Static IP	Optus infrastructure	Pacnet
Telstra	3G/4G	telstra.internet	Business account	This is the Telstra 10. private NATted network. Code is "GPTCOMB3".
Telstra	3G/4G	telstra.iph	Any account type	This is their "iPhone" APN. Call and ask for the "GPDWLES3" code to be added to your account.	Whirlpool
Telstra	3G/4G	telstra.extranet	Business account (10 digits)	This gives you a real, usable dynamic IP. Call and ask for the "GPTEXB3" code to be added to your account.
Telstra	3G/4G	telstra.corp	Business account (10 digits)	This is for Telstra Wireless IP WAN users. Call and ask for the "GPCORPB3" code to be added to your account.
Vodafone	3G/4G	live.vodafone.com		This is NATted on the 10.0.0.0/8 network	Vodafone
Vodafone	3G/4G	vfinternet.au		This gives you a real, usable dynamic IP.  Or so it appears.  But it is NATted through 10.64.64.64 still.  Ggrrrr...
Voicetalk	3G	splns357		Try getting any sense out of VoiceTalk!

There are some other sites out there with useful collections of APN configuration information:
Whirlpool APN - Access Point Name


Regards,

The Outspoken Wookie

Embedded "Security" with IPMI and UPnP

First, let me say that I've been outspoken about UPnP on gateway devices since UPnP was first released - it is simply a Bad Idea^TM.

Recently released information on an IPMI vulnerability involving UPnP on server motherboards has been published by Cari.net here.  Basically, it details how the BMC authentication details of almost 32,000 servers are available online, easily, in plain text - from the servers themselves.  Add to this the older Linux kernel versions some BMCs were running (any old version of any operating system will contain unpatched vulnerabilities that can be exploited for nefarious purposes) and you have a great recipe for easy and effective hacking of servers.

Not good.  Not good at all.

So, again, can I ask that people administering systems actually do their jobs properly and keep up to date with patches and updates and - particularly - disable vulnerable services from gateway devices and implement decent firewall rules to limit access to systems that are supposed to be protected behind these firewalls.


Regards,

The Outspoken Wookie

CPU Cores, NUMA Nodes and Performance Issues

I have a client who has been suffering from performance issues on a Remote Desktop Server guest that's running on a Hyper-V server.  I suppose some details may help here:

Original Configuration
Dell PowerEdge T410 Server
2 * Intel Xeon E5649 CPUs @ 2.53GHz
4 * 8GB 1333MHz DDR3 modules (32GB total)
Windows Server 2008 R2 Standard SP1 as the Hyper-V Host OS
 - Windows SBS 2011 (Server 2008 R2 SP1) as a Hyper-V Guest
 - Windows Server 2008 R2 SP1 (Remote Desktop and LOB Server) as a Hyper-V Guest

Current Configuration
Dell PowerEdge T410 Server
2 * Intel Xeon E5649 CPUs @ 2.53GHz
8 * 8GB 1333MHz DDR3 modules (64GB total)
Windows Server 2012 R2 Standard as the Hyper-V Host OS
 - Original Windows SBS 2011 (Server 2008 R2 SP1) as a Hyper-V Guest
 - Original Windows Server 2008 R2 SP1 as a Hyper-V Guest
 - New Windows SBS 2011 (Server 2008 R2 SP1) as a Hyper-V Guest (will replace original instance)
 - New Windows Server 2008 R2 SP1 (RDS) as a Hyper-V Guest (will replace original instance(1))
 - New Windows Server 2008 R2 SP1 (LOB) as a Hyper-V Guest
 - New Windows Server 2012 R2 (LOB) as a Hyper-V Guest

Now, we took this particular client over recently and they have been suffering various performance-related issues as well as LOB-related issues since the new system was installed (Aug-Sep, 2012). We'll just speak about the performance-related issues here...

This system has always been under-performing, sluggish and unstable. None of those are good things and we found a few causes for some of the issues, but realistically we felt the best result would be achieved by upgrading the RAM in the server and by rebuilding all the servers (software) and adding some more for application isolation purposes - we're not fans of doing what was originally done here (running LOB applications on an SBS 2011 box) or what was then tried as a fix (running LOB applications on a Remote Desktop Server). So, as Server 2012 R2 is the current Windows Server release, that's what we decided to run with - and also because its Hyper-V implementation gives us a lot more options such as live exports and much improved Hyper-V replication.

The one remaining major issue, after the RAM and Host OS upgrade, is still the sluggish performance of the original 2008 R2 RDS guest. It was topping its CPU out (in the guest) whilst barely using any host CPU resources (16%). Yes, the latest Hyper-V Integration Components are installed (for those wondering).

So, under the original Hyper-V Host (ie, 2008 R2), there were 4 Virtual Processors assigned to each guest, which is the maximum number of Virtual Processors that any guest can have under 2008 R2 Hyper-V.

Now, as this is a 2*6-core host (i.e. 12 real cores), which means a total of 24 Logical Processors including HyperThreading, we assigned 8 cores to the original RDS guest and 4 cores to the original SBS guest and moved on to other things such as building the new servers. Apparently, that's not all we needed to do - the SBS box was running fine using 16% of the host CPU resources however the RDS box was CPU-starved.

After a fair bit of investigation, fiddling, Googling, asking questions of people such as Kyle Rosenthal from WindowsPCGuy and general head scratching, hair pulling and frustration (all-round, from both the client and ourselves) I found the issue earlier this afternoon.

But first, things that COULD well have been the issue, but weren't:

1. I thought I bumped the CPU count up but hadn't
2. The host was actually flat-lining its CPU
3. I needed to install the latest Hyper-V Integration components
4. I needed to reinstall the latest Hyper-V Integration components over the top of the existing (latest) components (yet to find a way to actually achieve this)
5. I needed to uninstall and reinstall the latest components (again, yet to find a way to uninstall them)
6. I needed to drop the number of assigned logical processors from 8 back to 4, reboot, then bump from 4 to 8 and reboot again
7. I needed to drop back to a single logical processor, reboot, then up to 8 and reboot again

And now what I found to be the actual issue: "msconfig" seems to have been run in that 2008 R2 RDS virtual guest and then under Boot/Advanced, the # processors was limited to 4. I first thought about something like this after seeing that Device Manager showed all 8 virtual processors, but Task Manager/Perfmon only showed 4. So I had a look in "msconfig" and lo and behold - there was a limit of 4 CPUs set. I unchecked this option, rebooted and amazingly (well, OK, not really), all 8 CPUs were showing.

So, for good measure, I increased this to 12 virtual processors, rebooted again, and all 12 were showing in Task Manager. WOOHOO!!!

Once you go past 12 virtual processors (in this dual 6-core server), NUMA comes into play. NUMA (Non-Uniform Memory Access) is a way of allowing a processor (in hardware) or a virtual machine (in software) to access local memory faster than remote memory - in hardware, "remote" memory counts as memory connected directly to the bus of a different physical CPU. Now, NUMA not only comes into play when you start adding a large number of cores to a virtual guest, but also when you add more RAM than is physically available on one CPU to a guest - you get an approximately 20% performance hit when crossing a NUMA boundary. Because of this performance hit, you can actually get a performance reduction by adding too many logical processors and/or too much RAM to a virtual guest.

Microsoft has some information on NUMA that states (basically) the maximum memory in a NUMA node is the amount of physical RAM divided by the number of logical processors. That information seems to be rather outdated when you use current multi-core CPUs. If you're looking for some more updated information regarding NUMA node boundaries, I strongly suggest having a read of the article on Aidan Finn's blog found at http://www.aidanfinn.com/?p=11945 which refers to this blog post
http://www.benjaminathawes.com/2011/11/09/determining-numa-node-boundaries-for-modern-cpus/

Regards,

The Outspoken Wookie

Firewall with Hyper-V Synthetic NIC Support

I've been looking for a while to try and find an Open Source firewall for use in a Hyper-V environment. There's a number out there that will install and work, however trying to find a product with support for the Hyper-V Synthetic NICs instead of just the slower Legacy NICs has been... well... tedious.

I've looked at IPFire before - mainly as it was originally a fork of the IPCop project which is a fork of the SmoothWall project that I was on the development team of for some time. I was re-introduced to IPFire recently on a site visit to try and see if we could find causes for ADSL speed issues and then again at a friend's place last week who was running it in his home office.

So, on a whim, I thought I'd give it another go and see if it supported the Hyper-V Synthetic NICs on Hyper-V 2012 R2 and, well, kinda. During the installation, both the legacy NICs I added to the Hyper-V Guest were detected as was the default installed Synthetic NIC I'd not removed. So, I stopped the install, removed the Legacy NICs, added another Synthetic NIC and restarted the install. During installation, I chose the appropriate NIC for Green and Red, and off we went.

The first boot was great - the IPFire VM came up NICs blazing. :) I fiddled with the web interface a bit then rebooted. That's when it started to look like things weren't quite as I'd hoped - during the boot, I received an error message stating that "Interface green0 doesn't exist", however red0 worked fine. I re-ran "setup" and re-assigned the NICs and rebooted and this time "Interface red0 doesn't exist" was reported. Hhmmm...

So it seemed that Hyper-V Synthetic NICs were kinda supported. I had a look to see if the modules were being loaded properly and noticed a distinct lack of Hyper-V modules in /etc/sysconfig/modules. After a little Googling, I found the found the following information on the IPFire Install Guide:

Hyper-V

IPFire includes the modules required to work properly in a Hyper-V environment, but those modules are not enabled by default. To enable those modules, add the following four lines to the file /etc/sysconfig/modules and reboot:
hv_blkvsc
hv_netvsc
hv_storvsc
hv_vmbus

So, after adding these modules and rebooting a few times to test, all seems fine and IPFire is running with Hyper-V Synthetic NICs. :)

Now, for the speed testing results. I ran this test using 35.0GB of data consisting of some .iso files of around 4GB and also 7.8GB of smaller files of varying sizes (ie, extracted Windows Server 2012 R2 Standard and Windows Server 2012 R2 Essentials ISOs) with the following results.

Test 1 - Legacy NICs, across a 1GbE L3 switch from a physical server's USB-attached drive to this 2012 R2 Hyper-V Guest on a RAID-5 SSD Array on a 2012 R2 Standard server running Hyper-V

Test 2- Synthetic NICs, across a 1GbE L3 switch from a physical server's USB-attached drive to this 2012 R2 Hyper-V Guest on a RAID-5 SSD Array on a 2012 R2 Standard server running Hyper-V

Test 3- Synthetic NICs, across a 1GbE L3 switch from a physical server's SAS RAID-5 HDD Array to this 2012 R2 Hyper-V Guest on a RAID-5 SSD Array on a 2012 R2 Standard server running Hyper-V

Results:

Scenario	NIC Type	Source	Speed	Comments
Test 1	Legacy	Remote USB2	0.113GB/min	Yes, Legacy NICs are as slow as this!
Test 2	Synthetic	Remote USB2	1.62GB/minute	This is about the speed expected from USB2
Test 3	Synthetic	Remote SAS Array	4.32GB/minute	This is much more bearable!

So, it shows that the implementation of the Hyper-V Synthetic NIC drivers in IPFire definitely live up to expectations and provide much better performance than the old Legacy NICs can ever dream of.

Regards,

The Outspoken Wookie

Internet and WAN Connectivity - WTF are all these terms?

As we are all aware, the ICT (Information and Communication Technology) industry seems to exist solely to create new TLAs (Three Letter Acronyms) and ETLAs (Extended Three Letter Acronyms) and one of the most confusing places that these TLAs and ETLAs are used and abused is relating to networking in general and more specifically to Internet and WAN (Wide Area Internet) connectivity. So, I'll make an attempt here to help and clarify this a bit...

There are many ways to break Internet and WAN connectivity down, however there are two main types of Internet connection available today – asynchronous and synchronous. ADSL (Asynchronous Digital Subscriber Line), ADSL2 and ADSL2+ are all asynchronous connections – they have a faster download (inbound) speed than upload (outbound) speed. ADSL speeds are generally available up to 8192Kbps down and 384Kbps up and ADSL2+ speeds are generally available up to 24Mbps down and 1.2Mbps up. There’s also an ADSL2+ Annex M standard that is available in maximum speeds of around 20Mbps down and 3.2Mbps up. All these speeds are "best case" and basically will only be achieved if you are located in a building next to a telephone exchange, with these speeds dropping off as the distance from the exchange increases.

A business grade ADSL2+ service with 200GB or so of data is likely to cost in the vicinity of $100-$150 per month.

ADSL and ADSL2+ connections can also be configured to sacrifice a lot more of their download speed for increased upload speed, hence the availability of ADSL connections at 0.5Mbps/0.5Mbps and ADSL 2+ connections at 2Mbps/2Mbps. As you can see, these speeds are now synchronous, yet still delivered over one form of ADSL connection - way to help keep things clear...

Synchronous connections are delivered in 2 main formats – over copper or over fiber. Fiber connections can go faster (up to 1Gbps and higher), but the installation costs can be in the region of $5000. Synchronous connections over copper are generally available up to 40Mbps and are often referred to as “Ethernet in the First Mile” (EFM), “Ethernet Over Copper” (EOC) and “Mid-Band Ethernet” (MBE) and these terms are, to all intents and purposes, interchangeable.  Installation costs on an EOC connection are in the region of $1200.

A 10Mbps EOC connection with 200GB or so of data is likely to cost in the vicinity of $250-$500 per month, and on Fiber this will likely cost around $500-$750 per month depending on the service provider and the distance from the exchange.

Historically, asynchronous connections were more than acceptable for most individuals and businesses as most of the time people were *downloading* things like files and web pages from the Internet and rarely *uploading* information, but as time has progressed this has become more the exception than the norm for many businesses as they are using online storage for documents, photos and so on, online email and groupware servers and connections between multiple offices. This is where synchronous connections have become more popular for businesses.

The National Broadband Network (NBN) here in Australia is a bit of an odd one out. It is an asynchronous connection, but delivered at speeds of up to 100Mbps down and 40Mbps up – so it delivers very decent outbound speeds, with even faster inbound speeds. Of course, this is for those lucky enough to have had this rolled out before the Federal Liberal/National Party decided that high speed Internet was too scary - people get easy access to educational material - and destroyed Australia's chance at decent Internet speeds.

A 100/40Mbps NBN connection with 200GB or so of data is likely to cost in the vicinity of $120-$170 per month, depending on the service provider and extras included in the plan.

Now, just because you have an ADSL2+ or EOC connection doesn't necessarily mean it is connected directly to the Internet. Many larger, geographically diverse businesses will have an ADSL2+, Fiber or EOC tail connected to their multiple locations that are all brought back into their ISP's network and from there, connected to the Internet. This is the "WAN" part of the "Internet and WAN" in the article title. These sorts of connections are often referred to as an MPLS (Multi-Protocol Label Switching) Network, VPLS (Virtual Private LAN Service) Network or simply a Private Network.

Another way to interconnect multiple locations is across the Internet using a VPN (Virtual Private Network). This is where each site is connected directly to the Internet and across this Internet connection is run a secure pipe (the VPN) that connects the multiple sites. Types of VPN connection include EoIP (Ethernet over Internet Protocol), IPSEC (Internet Protocol Security), L2TP (Layer 2 Tunneling Protocol), PPTP (Point-To-Point Tunneling Protocol) and SSTP (Secure Sockets Tunneling Protocol).

Another term you may have heard is "Contention Ratio". What this means, basically, is the number of customers of a particular ISP (Internet Service Provider) who are sharing the bandwidth that you are paying for. So, if you see a 1:1 contention ratio, this means that the speed of the connection you are paying for is reserved for you into the ISP's core network. A contention ratio of 4:1 means that you're sharing that bandwidth into the ISP's core network with 3 other customers. Residential contention ratios are significantly higher than those for business customers, which is one of the reasons that residential connections are priced lower than business grade connections.

Finally, you may have heard the terms "Layer 2 Connection" or "Layer 3 Connection" which are a little more complex to explain as compared to the previous terms, however the simple way to look at it is a L3 Connection will have all traffic from the ISP's network sent to you at a single QoS (Quality of Service) level - you can't ask for inbound VoIP (Voice over Internet Protocol) to have higher priority than general web browsing, email, file downloading or anything else. A L2 Connection will allow you to have inbound traffic prioritized from the ISP to your network, so a large inbound email won't stomp on a VoIP call from a potential client. This may help explain why L2 Connections are often a little more expensive than a L3 Connection.

So, back to the big question - what do I want and when do I want it?

If your Internet requirements don't involve a lot of VoIP, video conferencing, Private Network, VPN or general outbound traffic, an ADSL2+ service will likely suit quite nicely. However, if you do utilise VoIP, VPNs to other sites or remote workers, video conferencing or have various cloud-based services such as hosted email, hosted file services or have a requirement for a better SLA (Service Level Agreement) than "we'll try to get it running again... sometime", some form of EoC or Fiber network connectivity (Internet or WAN) would likely be a better option.


Regards,

The Outspoken Wookie

pfSense in Hyper-V 2012 R2

As of May 2012, Microsoft has supported FreeBSD running as a guest on Hyper-V (see this article for more info).  That's nice as pfSense runs on a FreeBSD base, and if all was well in the world, the recently released pfSense 2.1 would have supported these new drivers.  If.

Unfortunately, pfSense 2.1 doesn't include the required drivers, so we're still stuck with Legacy NICs.  :(  Oh, well...

So, if you want to configure a pfSense Hyper-V 2012 R2 guest, you'll have to stick with the 100Mbps limitation of the Legacy NICs and a little bit of time synchronization funkiness due to the Hyper-V Host CPUs entering into low power mode and pfSense not handling this all that well, resulting in a number of "calcru: runtime went backwards" error messages.  :(

So, at this point in time pfSense 2.1 works adequately for a testing environment under Hyper-V, but I wouldn't recommend using it for a production environment.

1. The latest pfSense is available from: http://mirror.optus.net/pub/pfSense/downloads/ - choose the LiveCD-x.y-RELEASE-amd64.iso.gz or LiveCD-x.y-RELEASE-i386.iso.gz file, check its checksum after downloading, and extract the ISO image
2. Create a Gen 1 Hyper-V Guest with one CPU, 512MB RAM, 2 * Legacy NICs (and no Synthetic/native ones) and disable the Time Synchronization option.  Make a 5GB or so fixed VHDX file and assign the ISO as the DVD.  Boot away
3. After the LiveCD boots and the two NICs (de0 and de1 have been assigned), you have the option to install to HDD - take this option and remove the ISO after the install and before the reboot happens
4. Ensure the IPs of the two interfaces are configured appropriately.  I configured de0 to connect to the physical interface and de1 to connect to a Private Network for the guests inside the pfSense firewall.  Check that you can ping 8.8.8.8 from the console.
5. Configure a guest on the Private Network, check it can ping 8.8.8.8 and www.google.com
6. Hit the pfSense web page from inside the network and configure any options you need.
7. On the pfSense console, you may need to type the following to ensure the NICs are restarted properly.  This used to be a significant issue with earlier pfSense releases, however it seems to have been fixed in 2.1 - YMMV:
   echo "ifconfig de0 down" >> /etc/rc.local
   echo "ifconfig de0 up" >> /etc/rc.local
   echo "ifconfig de1 down" >> /etc/rc.local
   echo "ifconfig de1 up" >> /etc/rc.local
8. To try and help a little with the time sync issues, you will likely also need to type:
   echo "sysctl kern.timecounter.hardware=TSC" >> /etc/sysctl.conf
9. That's pretty much it.  You'll have a somewhat functional pfSense Hyper-V guest.  It would be nice if the pfSense team had incorporated the Hyper-V drivers - let's hope they actually do this for pfSense 2.2.

Regards,

The Outspoken Wookie

Custom Outlook.com and Windows Live Domains

Windows Live Admin Center

Changes to custom domains with Outlook.com

Outlook.com no longer offers support for new custom domain sign ups. New customers looking to manage custom domains are encouraged to use Office 365, Microsoft's premium online service, which also includes enterprise-class mail, collaboration and communication tools.

Explore Office 365 subscriptions

If you currently use custom domains with Outlook.com

We will continue to support the ability to log in to Outlook.com with your existing custom domain email address, but in the future, you will no longer be able to add or remove accounts in your domain.

For domain administrators who wish to continue to manage email addresses in their domain, we will be providing the option to migrate your account to Office 365. As part of Office 365, you will be able to continue managing your custom domains, as well as use many additional productivity tools including shared calendars, cloud storage and instant messaging.

Current customers using custom domains with Outlook.com will be notified via email in the coming weeks with further details and instructions on how to migrate.

----------

What a complete bunch of bastards!!!  :(

Regards,

The Outspoken Wookie

Making Google the Default Search Provider in IE10 and IE11

I hate that you can't do this directly in Internet Explorer as Microsoft wants us to use the appalling Bing search (sic) engine.  So, instead, just click on http://www.iegallery.com/en-US/Addons/Details/813 and add this as a Search Engine, then go to Manage Add-Ons and ensure the "Search in the address bar" checkbox is checked (should be), change the listing order to make Bing feel even more unloved and your searches in IE will finally be functional! :)

UPDATE: Microsoft has made it even harder to make Google the default search engine.  You nowe have to go to the URL above, which Microsoft will redirect, then you need to go to the URL again for the Google Search option to appear.  Go Microsoft - you bastards!

Regards,

The Outspoken Wookie