finally, after a dreaded long battle, I have found the culprit, or atleast something that is going to lead me directly to it on why the site loads slow initially, but renders fast (due to Cache) afterwards. I have installed glimpse, and now able to see a Trace in the Tabs of Glimpse, that is telling me that Umbraco Reserved Urls are taking approximately 37 seconds to respond to a page load. Image below:

So, looking at my web.config now, shows this for Umbraco Settings:

<add key="umbracoDbDSN" value="server=127.0.0.1;database=DB;user id=sa;password=password" />
<add key="umbracoConfigurationStatus" value="4.6.1" />
<add key="umbracoReservedUrls" value="~/config/splashes/booting.aspx,~/install/default.aspx,~/config/splashes/noNodes.aspx" />
<add key="umbracoReservedPaths" value="~/umbraco,~/install/,~/testbed/,~/floorplans/,~/" />
<add key="umbracoContentXML" value="~/App_Data/umbraco.config" />
<add key="umbracoStorageDirectory" value="~/App_Data" />
<add key="umbracoPath" value="~/umbraco" />
<add key="umbracoEnableStat" value="false" />
<add key="umbracoHideTopLevelNodeFromPath" value="true" />
<add key="umbracoEditXhtmlMode" value="true" />
<add key="umbracoUseDirectoryUrls" value="true" />
<add key="umbracoDebugMode" value="true" />
<add key="umbracoTimeOutInMinutes" value="20" />
<add key="umbracoVersionCheckPeriod" value="7" />
<add key="umbracoDisableXsltExtensions" value="true" />
<add key="umbracoDefaultUILanguage" value="en" />
<add key="umbracoProfileUrl" value="profiler" />
<add key="umbracoUseSSL" value="false" />
<add key="umbracoUseMediumTrust" value="false" />
<add key="umbracoContentXMLUseLocalTemp" value="false" />

Please note: umbracoReservedPaths has ~/ in it, but if I remove it, the pages of my site come up blank. I also have UrlRewrite Module installed and looks like this in the web.config:

In httpModules section of system.web:

<add name="Glimpse" type="Glimpse.AspNet.HttpModule, Glimpse.AspNet" />
<!-- URL REWRTIER -->
<add name="UrlRewriteModule" type="UrlRewritingNet.Web.UrlRewriteModule, UrlRewritingNet.UrlRewriter" />
<add name="umbracof" type="umbraco.presentation.requestModule" />
<!-- UMBRACO -->
<add name="viewstateMoverModule" type="umbraco.presentation.viewstateMoverModule" />
<add name="umbracoBaseRequestModule" type="umbraco.presentation.umbracobase.requestModule" />
<!-- ASPNETAJAX -->
<add name="ScriptModule" type="System.Web.Handlers.ScriptModule, System.Web.Extensions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
<!-- CLIENT DEPENDENCY -->
<add name="ClientDependencyModule" type="ClientDependency.Core.Module.ClientDependencyModule, ClientDependency.Core" />
<add name="Redirector" type="Redirector" />

In system.webServer I have the following:

<validation validateIntegratedModeConfiguration="false" />
<modules runAllManagedModulesForAllRequests="true">
    <remove name="Session" />
    <remove name="ScriptModule" />
    <remove name="UrlRewriteModule" />
    <remove name="umbracoRequestModule" />
    <remove name="viewstateMoverModule" />
    <remove name="umbracoBaseRequestModule" />
    <remove name="ClientDependencyModule" />
    <add name="Glimpse" type="Glimpse.AspNet.HttpModule, Glimpse.AspNet" preCondition="integratedMode" />
    <add name="UrlRewriteModule" type="UrlRewritingNet.Web.UrlRewriteModule, UrlRewritingNet.UrlRewriter" preCondition="" />
    <add name="umbracoRequestModule" type="umbraco.presentation.requestModule" />
    <add name="viewstateMoverModule" type="umbraco.presentation.viewstateMoverModule" />
    <add name="umbracoBaseRequestModule" type="umbraco.presentation.umbracobase.requestModule" />
    <add name="ScriptModule" preCondition="managedHandler" type="System.Web.Handlers.ScriptModule, System.Web.Extensions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
    <add name="ClientDependencyModule" type="ClientDependency.Core.Module.ClientDependencyModule, ClientDependency.Core" />
    <add name="Redirector" type="Redirector" />
    <add name="Session" type="System.Web.SessionState.SessionStateModule" preCondition="managedHandler" />
</modules>

Ofcourse there is much more, but this is the <modules> tag in there.

In <configuration> <configSections> I have the following:

<section name="urlrewritingnet" restartOnExternalChanges="true" requirePermission="false" type="UrlRewritingNet.Configuration.UrlRewriteSection, UrlRewritingNet.UrlRewriter" />

Is this not configured properly? Why is Umbraco Reserved Urls taking 37 seconds to respond? Doesn't make much sense to me.