A look at win32 virtual address for a process and its creation.
Lets begin by reviewing the steps involved in creating a process using a call such as CreateProcess()
First the image file (.exe) to be executed is loaded inside the creating process (caller)
Second the executive process object is created.
Third the initial thread (stack, context and executive thread object) is created
Then the Wn32 subsystem is notified of the new process so that it can set up for the new process and thread.
Then the calling process can start execution of the initial thread (unless the Create Suspended flag was specified) and return
 In the context of the new process and thread, the initialization of the address space is completed ( such as the load of the required DLLs) and then the execution at the entry point to the image is started. ( main subroutine )
Its probably relevant to mention here that in the first stage of loading the image, we find out what kind of application it is. If it's a Win32 application, it can be executed directly. If it's a OS/x we run the OSx.exe. If its POSIX we run the Posix.exe and if its MSDOS we run the MS-DOS directly.
In the second stage, we create the executive process object.  This involves the following substages:
Setting up the EPROCESS block
Creating the initial process address space
Creating the kernel process block
Concluding the setup of the process address space
Setting up the PEB
and completing the setup of the executive process object.
In the substages above, the one involving the creation of the initial Process Address Space will consist of three pages:
the Page directory
the Hyperspace page and
the Working set list
In the sixth stage which perform process initialization in the context of the new process, the image begins execution in user mode. This is done by creating a trap frame that specifies the previous mode as a user and the address to return to as the main entry point of the image. Thus, when the the trap that causes the thread to start execution in kernel mode is dismissed, it begins execution in user mode at the entry point.
Inside the x86 process virtual address space starting from low to high memory, the kernel portion at the high memory consists of HAL usage sections, crash dump information, nonpaged pool expansion, system PTEs, paged pool, system cache, system working set list, etc. The user mode consists of hyperspace and process working set list, page tables and page directory, system PTEs, mapped views and system code.  The kernel and the user mode addresses are separated by unused areas. In Linux, we have a memory mapped region in this space. The kernel space is reserved in the higher memory. The stack grows downwards adjacent to kernel space. The heap grows upwards from the user space.
Note that the instruction set is static and so we have

    if (pFunc->get_addressSection ( &seg ) == S_OK &&
        pFunc->get_addressOffset ( &offset ) == S_OK)
    {
        pFunc->get_length ( &length );
        pSession->findLinesByAddr( seg, offset, static_cast<DWORD>( length ), &pEnum );
    }

IDiaSymbol* pFunc;
pSession->findSymbolByAddr( isect, offset, SymTagFunction, &pFunc ); 

Today we will discuss how to use the DIA interfaces for reading the PDB when compared to our previous post here
MSDN says the following steps:

//initialize

CComPtr<IDiaDataSource> pSource;
hr = CoCreateInstance( CLSID_DiaSource,
                       NULL,
                       CLSCTX_INPROC_SERVER,
                       __uuidof( IDiaDataSource ),
                      (void **) &pSource);

if (FAILED(hr))
{
    Fatal("Could not CoCreate CLSID_DiaSource. Register msdia80.dll." );
}

//load

wchar_t wszFilename[ _MAX_PATH ];
mbstowcs( wszFilename, szFilename, sizeof( wszFilename )/sizeof( wszFilename[0] ) );
if ( FAILED( pSource->loadDataFromPdb( wszFilename ) ) )
{
    if ( FAILED( pSource->loadDataForExe( wszFilename, NULL, NULL ) ) )
    {
        Fatal( "loadDataFromPdb/Exe" );
    }
}

CComPtr<IDiaSession> psession;
if ( FAILED( pSource->openSession( &psession ) ) ) 
{
    Fatal( "openSession" );
}

CComPtr<IDiaSymbol> pglobal;
if ( FAILED( psession->get_globalScope( &pglobal) ) )
{
    Fatal( "get_globalScope" );
}

CComPtr<IDiaEnumTables> pTables;
if ( FAILED( psession->getEnumTables( &pTables ) ) )
{
    Fatal( "getEnumTables" );
}

CComPtr< IDiaTable > pTable;
while ( SUCCEEDED( hr = pTables->Next( 1, &pTable, &celt ) ) && celt == 1 )
{
     // Do something with each IDiaTable.
}

And the following lines indicate how to translate addr to function lines:

void dumpFunctionLines( IDiaSymbol* pSymbol, IDiaSession* pSession )
{
    ULONGLONG length = 0;
    DWORD isect = 0;
    DWORD offset = 0;
    pSymbol->get_addressSection( &isect );
    pSymbol->get_addressOffset( &offset );
    pSymbol->get_length( &length );
    if ( isect != 0 && length > 0 ) {
        CComPtr< IDiaEnumLineNumbers > pLines;
        if ( SUCCEEDED( pSession->findLinesByAddr( isect, offset, static_cast<DWORD>( length ), &pLines ) ) ) {
            CComPtr< IDiaLineNumber > pLine;

:

:

Today   I want to talk about auditing in the context of software. Auditing is not only for security but also for compliance and governance. Typically they are used for hashing or signing as well as to create an audit trail. When different accounts access a protected resource, an audit trail can reconstruct the timeline of actors and their operations. This is done with the help of two keys a private key and a public key. The private key is for encrypting a message so that the message is not transparent.  Anyone with a public key  though can decrypt it. Of course the public key has to be shared in a trustworthy way and hence a certification authority is involved as an  intermediary. A receiver with a public key can then decrypt the message back to its original content.
Unlike other products functionality features auditing has a special requirement. It is said that incomplete security is worse than no security. The main concern here is how to audit events and sign them such that they have not been tampered with. Auditing explains 'who' did 'what' 'when'. As such it must be pristine. If there is tampering, it lays waste all the information given so far. Moreover, information for a specific event may be critical. If this information is tampered with, there could be a different consequence than the one intended. Even though audit events are generally not actionable except for regulations and compliance, it is another data point and is often demanded from software systems. Hence auditing is as important a feature as any, if not more. To add to the requirements of auditing,  one important requirement is that security events may also be a consumer of auditing. Information about actions, actors and subjects are also events and as such are met with the same signing and hashing that auditing does.

One of the concerns with the sensitivity of the information is addressed by preventing any changes to audited event such as update or delete. This we see is helpful to guaranteeing that the data has not been tampered with. But can it be enforced. what mechanism can we put in place for this ?
Some techniques include hashing. A hashing function such as SHA256 for example will generate the same hash for the same input. Moreover the input may not be deciphered from its hash. Given a hash and the original from different sources, we can now check whether they are the same.
Another technique is to use the public key cryptography mentioned above. A public key and a private key can be used to enable only a selected set of receivers to understand the message. The public key is distributed while the private key is secured. When the public key is handed out, it is assumed to come from the sender that is why it is called a certificate.
If you would like to verify that the certificate did come from the sender, we can reach the certificate authority.The private key is left to the issuing party to secure.The keys can be generated with tools but its left for the party to secure. That said, if the users for the software do not interact with the keys generally or are admins themselves, then this is less severe.
In general its good to be a little paranoid about systems when it comes to security. 

Tonight I will discuss some more on function pointer to symbol translation using displacement from base.  because we found that such translation seems difficult without SymInitialize method that takes the handle of the target process. We will look at getting that information from dumps. MSDN says that the handle passed to SymInitialize must be the same value passed to all other symbol handlers functions called by the process. It is the handle that the functions use to identify the caller and locate the correct symbol information. Only that a process that acts like a debugger should not call the GetCurrentProcess to provide the handle and instead provide the handle of the target process. Thus the handle merely seems to be a token for a context. The other parameters to this function are UserSearchPath and fInvadeProcess. The UserSearchPath are the series of paths to be searched for a symbol file. This is an optional parameter. The fInvadeProcess when enumerates the loaded module for the process.
When you try out the SymInitialize with a bogus non-zero handle and follow it with a SymFromAddr call, it returns the error indicating that the handle is invalid.
Now we could use the

DWORD64 WINAPI SymLoadModuleEx(
  _In_  HANDLE hProcess,
  _In_  HANDLE hFile,
  _In_  PCTSTR ImageName,
  _In_  PCTSTR ModuleName,
  _In_  DWORD64 BaseOfDll,
  _In_  DWORD DllSize,
  _In_  PMODLOAD_DATA Data,
  _In_  DWORD Flags
);

but that method also doesn't help us specify the handle of a non-existent process.
So the only option left is to try creating a process with CreateProcess and DEBUG_PROCESS flag to specify that the created process is in debugging mode.

BOOL WINAPI CreateProcess(
  _In_opt_     LPCTSTR lpApplicationName,
  _Inout_opt_  LPTSTR lpCommandLine,
  _In_opt_     LPSECURITY_ATTRIBUTES lpProcessAttributes,
  _In_opt_     LPSECURITY_ATTRIBUTES lpThreadAttributes,
  _In_         BOOL bInheritHandles,
  _In_         DWORD dwCreationFlags, <------------------------------------- DEBUG_PROCESS
  _In_opt_     LPVOID lpEnvironment,
  _In_opt_     LPCTSTR lpCurrentDirectory,
  _In_         LPSTARTUPINFO lpStartupInfo,
  _Out_        LPPROCESS_INFORMATION lpProcessInformation
);

So I tried out with :

// Raw2Sym.cpp : Defines the entry point for the console application.

//

#include "stdafx.h"

#include "windows.h"

#include "dbghelp.h"

#pragma comment(lib, "dbghelp.lib")

int _tmain(int argc, _TCHAR* argv[])

{

//HMODULE hDbgModule = LoadLibrary(L"dbghelp.dll");

if (argc < 3) 

{

printf("Usage: raw2sym base_addr func_ptr func_ptr ...\r\n");

return -1;

}

DWORD  error = 0;

HANDLE hProcess = 0;

bool initialized = false;

hProcess = GetCurrentProcess();

printf ("hProcess = %ld\r\n", hProcess);

_TCHAR wszPath[MAX_PATH+1] = {0};

WCHAR  szImageName[MAX_PATH] = TEXT("foo.exe");

DWORD64 dwBaseAddr = _tcstoui64(argv[1], NULL, 16);

printf("Base Address: %p\r\n", dwBaseAddr);

HANDLE hExe = 0;

HANDLE hFile = 0;

if (!GetCurrentDirectory(MAX_PATH, wszPath))

{

error = GetLastError();

printf("GetCurrentDirectory returned error: %d\r\n", error);

goto exit;

}

else

{

wprintf(L"Current directory is %s\r\n", wszPath);

}

STARTUPINFO si;

        PROCESS_INFORMATION pi;

        ZeroMemory( &si, sizeof(si) );

        si.cb = sizeof(si);

        ZeroMemory( &pi, sizeof(pi) );

        if(!CreateProcess(L"foo.exe", NULL, 

    NULL, NULL, false,DEBUG_PROCESS, NULL, wszPath, &si, &pi))

{

error = GetLastError();

printf("CreateProcess returned error: %d\r\n", error);

goto exit;

}

hExe = pi.hProcess;

if(!hExe)

{

printf("raw2sym must have splunkd exe and pdb in the same folder as the tool.\r\n");

return -1;

}

else

{

printf ("hExe = 0x%x\r\n", hExe);

}

if (!SymInitializeW(hExe, wszPath, FALSE))

{

// SymInitialize failed

error = GetLastError();

printf("SymInitialize returned error : %d\r\n", error);

goto exit;

}

else

{

initialized = true;

}

        SymSetOptions(SYMOPT_UNDNAME | SYMOPT_DEFERRED_LOADS);

hFile = CreateFile(L"foo.pdb", 0, 0, NULL, 0, 0, NULL);

DWORD64 loadedAddress = SymLoadModuleEx(hExe, hFile, "foo.pdb", NULL,dwBaseAddr , 0, NULL, SLMFLAG_VIRTUAL);

if(loadedAddress == 0)

{

// SymLoadModuleEx failed

error = GetLastError();

printf("SymLoadModuleEx returned error : %d\n", error);

goto exit;

}

else

{

printf("Loaded Address is %p: \r\n", loadedAddress);

}

for (int i = 2 ; i < argc; i++)

{

DWORD64  dwDisplacement = 0;

DWORD64  dwAddress = _tcstoui64(argv[i], NULL, 16);

char buffer[sizeof(SYMBOL_INFO) + MAX_SYM_NAME * sizeof(TCHAR)];

PSYMBOL_INFO pSymbol = (PSYMBOL_INFO)buffer;

pSymbol->SizeOfStruct = sizeof(SYMBOL_INFO);

pSymbol->MaxNameLen = MAX_SYM_NAME;

if (!SymSetScopeFromAddr(hProcess, loadedAddress))

{

DWORD error = GetLastError();

printf("SymSetScopeFromAddr for %s returned error : %d\r\n", argv[i], error);

}

if (SymFromAddr(hProcess, dwAddress, &dwDisplacement, pSymbol))

{

printf("%s\r\n", &(pSymbol->Name));

}

else

{

// SymFromAddr failed

DWORD error = GetLastError();

printf("SymFromAddr for %s returned error : %d\r\n", argv[i], error);

}

}

exit:

if (initialized && hExe)

SymCleanup(hExe);

if (hExe)

CloseHandle(hExe);

if (hProcess)

CloseHandle(hProcess);

if (hFile)

CloseHandle(hFile);

return 0;

}

But that returns an error:

c:\downloads\Raw2Sym\x64\Release>Raw2Sym.exe 000007f79b1b0000 000007f79b1bbb90

hProcess = -1

Base Address: 000007F79B1B0000

Current directory is c:\downloads\Raw2Sym\x64\Release

hExe = 0x34

Loaded Address is 000007F79B1B0000:

SymSetScopeFromAddr for 0 returned error : 6

SymFromAddr for 0 returned error : 6

Error code: (Win32) 0x6 (6) - The handle is invalid.

Instead found a helpful resource here : http://www.debuginfo.com/examples/dbghelpexamples.html
and giving it a try. The code mentioned under SymLoadFromPdb.cpp gives some suggestions but doesn't work either.

c:\downloads\Raw2Sym\x64\Release>Raw2Sym.exe c:\downloads\Raw2Sym\x64\Release\splunkd.pdb 00

1b0000 000007f79b1bbb90

Loading symbols for: c:\downloads\Raw2Sym\x64\Release\splunkd.pdb ...

Error: SymLoadModule64() failed. Error code: 0

A better option might be to use DIA sdk instead as mentioned here : http://msdn.microsoft.com/en-us/library/4sy4ebfw.aspx .
 Conceptually, I don't know if any technique will allow this method since the process/dump needs to be available in memory or disk to associate the base address and the offset. The instruction set is unchanged after the compile so the pdb and exe will be the same. Resolving the symbols from the instruction doesn't vary
As Dia SDK shows here in the article querying the pdb: http://msdn.microsoft.com/en-us/library/eee38t3h.aspx

Software support folks in the field often encounter crash dumps at the customer's premise. There they have some dumps and they don't have any access to symbols particularly when the customer's operating system is Windows.What they see in the dumps or crash log reports are stack frames with raw addresses such as 0x7ff34555 and these are not yet resolved as module!class::method. To investigate the issue, they need to resolve the stackframes. To do this sometimes they have to open the crashdump with a debugger. But even the debugger needs the symbols to resolve the stack frame and the support folks can't take the symbols to the customer's machine. So there is a round-trip involved in taking the dumps to the developers because the symbols and the raw stacktrace are not together. Resolving raw function pointers to symbols in an offline manner is therefore desirable.
This can be achieved by using the same module that the debuggers use to translate the stack trace - dbghelp.dll. This module exports methods such as StackWalk and StackWalk64 that gives the stack trace when pointed to the dump. But more interestingly it has methods SymFromAddr and SymFromAddrW that just seems to do what we want. To use this method we call SymInitialize first so that we have a handle to the process. The method takes the following parameters :
- a handle retrieved from the SymInitialize method
- an Address pertaining to a frame in the stack for which the symbol should be located.
- a displacement from the beginning of a symbol or zero and this is optional
- a buffer to receive the SYMBOL_INFO resolved from the address.
If the function succeeds, it returns true. The SYMBOL_INFO struct has a member called MaxNameLen that should be set to the size of the name in characters. The name can be requested to be undecorated for easy readability.
The code from MSDN is something like this:

DWORD64  dwDisplacement = 0;
DWORD64  dwAddress = SOME_ADDRESS;

char buffer[sizeof(SYMBOL_INFO) + MAX_SYM_NAME * sizeof(TCHAR)];
PSYMBOL_INFO pSymbol = (PSYMBOL_INFO)buffer;

pSymbol->SizeOfStruct = sizeof(SYMBOL_INFO);
pSymbol->MaxNameLen = MAX_SYM_NAME;

if (SymFromAddr(hProcess, dwAddress, &dwDisplacement, pSymbol))
{
    // SymFromAddr returned success
   printf("%s\n", buffer);
}
else
{
    // SymFromAddr failed
    DWORD error = GetLastError();
    printf("SymFromAddr returned error : %d\n", error);
}

When we give some address it should be the relative offset from the base address.

In this post we will continue to look at logparser and Splunk. Consider a case when Splunk can translate searches to SQL queries. This would mean that all the operators that Splunk enables on the search bar such as regex, rex, top, stats, multikv, collect etc all work without seeing any difference between Splunk indexes or SQL data providers.  Splunk seamlessly parses the search operations on data without importing it into its indexes. In such cases there would have to be a translation of Splunk search operators into LINQ or SQL depending on where the data resides. A cursory look at the operators will suggest that the predicates be pushed down as close to the data as possible. In this case, Splunk keeps its index and data operators as close as possible. If the operators were to target the data on an external source there would be several copies of the data and translations involved. This will be similar to the pipe operation in Splunk. Splunk exposes several semantics that work well in pipe operations. This is very helpful to IT world for administration as well as for automation. What Splunk provides for analysis is significantly improved by its search language. While we can generalize the same operators for other data sources, the search language works well for Splunk data because of the fields extraction and event indexing.

LINQ on Splunk

Language-Integrated Query (LINQ) is a set of features introduced in Visual Studio 2008 that extends powerful query capabilities for use by applications. LINQ is widely used in web applications and services in a variety of industries such as finance, retail, telecommunication and insurance. LINQ provides two features. It enables constructs in the language that have become very popular with web developers. And it abstracts the underlying data source while enabling the same query logic to operate against any data source. The language syntax for querying are available as standard query operators and they are very much in nature to SQL ANSI standard which have an established foundation in data storage and retrieval. While SQL is ubiquitously used for relational data, the standard query operators are not restricted to work against a database. LINQ can be executed on XML, plaintext, CSV, databases etc. so long as the standard query operators can see a sequence of objects on which to perform their operations such as selection, conversion, merging etc. This enumeration of the items in a sequence is defined by an interface popularly known as IEnumerable.  The behavior demanded from collections implementing this interface is that the items can be extracted via iterations. At the same time, the data providers implement an interface called the IQueryable that can take an expression tree and execute it on the data source. Together these interfaces connect an application to the data source with the help of powerful querying capabilities.

Splunk also provides programmable interfaces to its data source via what are known as Entities and Collections, which provide a common mechanism for a plethora of resources. Most of the SDK makes REST API calls and hence operates on a single resource or a collection. Entity and EntityCollection both derive from common base class Resource.  Note that this pattern exposes the resources per se and not their querying capabilities.  When objects are instantiated they are local snapshots of values as read and copied from the server. Changes made on the server are not available until a refresh is called. Compare this with the object relational mapping (ORM) software that use LINQ patterns. The ORMs are able to interpret that the state of an object has become ‘dirty’ from local updates and needs to save to disk seamlessly.  At the same time, an observer pattern notifies the application of the updates made to the server data.

Splunk does have its own powerful querying capabilities.  Splunk search operators are similar to Unix style piped operators and are also available for use from the same SDK. These are exposed as searches and jobs and they can take a string containing the search query and execute it on the server. The results are then returned to the caller. Searches can be either one shot or real time.  The one-shot is a synchronous API and it returns a stream. Real-time searches return live events as they are indexed, and this type of search continues to arrive. To view the results from a real-time search, we view the preview results.  Some parameters to the Search Job enable this mode. In the normal execution mode, a Search Job is first created with a search string and then the job is polled for completion. A built-in XML parser can then render the stream of results.

Thus we note that there is a mismatch in the way the items are retrieved from the server from the server to the application between LINQ and Splunk search. That said, the power of Splunk can be made available via adapters via LINQ patterns.