How I have tested EndPointProtection solution...

             Just finished evaluation for some EPP products. And I found difficult this challenge: how to chose solution that will fit your organisation. Of course first you need to understand if it is reasonable to spend money for EPP, probably you can get same or better results without EPP (like better app-control,  DeviceGuard and etc). But if you sure, that you need it, and it is the only one way for you, then what you gonna do? Right answer - run PoC/pilot with chosen vendors, and check what YOU as a customer need. Sales demos and presentations, NSS lab reports and Gartner qaudrants - will not help much. I want to share just how I did it. (it is not a best or full way, I am working not in SECURITY company, so it just a way how I can differ one solution from another as a customer, from techincal point of view).  This text is only about exploits/attacks protection quality, and I do not want to cover too much other things in this blog-post, but  they are also important, for example: classification of collected data (by vendor/solution, if we have deal with cloud based solutions), report quality, SIEM integration, customization, perfomance, attack surface (yes, even EPP making your attack surface bigger, like any AV...) and etc... 

C

          

       1.      My test methodology (exploit/attack/malware protection quality)

Pic 1. Kill Chain stages that are chosen by us for testing.

 1.1        Coverage and quality

1.1.1       Delivery

I have prepared few different exploits for testing:

a)     HTML file with Buffer Overflow exploit in browser plugin (0day)

b)    HTML file with Use-After-Free exploit in browser plugin (0day)

c)     DOC file with malicious macros (0day)

d)    EXE file with known 1day exploit for local privilege escalation (ring0 exploit)

e)     EXE files with known malware

a.      Default: known hash

b.     Encoded: unknown hash

f)      DLL files with known malware

TEST CASE	EXPECTED BEHAVIOR
DELIVERY: HTML with exploits	Detected and blocked
DELIVERY: WORD with macros	Detected and blocked
DELIVERY/INSTALL: Encoded Meterpereter x64 as an EXE drop	Detected and blocked
DELIVERY/INSTALL: Encoded Meterpereter x64 as a DLL drop	Detected and blocked
DELIVERY/INSTALL: Hash based malware	Detected and cleaned
DELIVERY/INSTALL: Not Hash based malware	Detected and blocked

1.1.2       Exploitation

For simulating 0day attacks/exploits I have created own vulnerable Internet Explorer plugin and wrote exploits for it. Same for DOC file tests on delivery stage, I have created special, obfuscated macros. Because exploitation step is the most critical step, I did analysis how each security solution under test detects and prevents exploitation of vulnerabilities, and accordingly evaluated the product per the defined expected behavior.

TEST CASE	EXPECTED BEHAVIOR
EXPLOITTAION: ROP Shellcode	Blocked on any stage of ROP shellcode
EXPLOITTAION: Shellcode	Blocked on any stage of normal shellcode
EXPLOITTAION: 1day ring0 exploit	Blocked on any stage
EXPLOITTAION:  HeapSpray	Detected or blocked
EXPLOITTAION: Exploit triggers - UAF, Bof	Detected or blocked
EXPLOIT: 0day Macros run	Blocked
EXPLOIT: Meterpreter  x86 (in mem)	Detected or Blocked
EXPLOIT: IOC bypass - migration to EXPLORER (CreateRemoteThread)	Detected or Blocked

1.1.3       Installation

Have used self-made (unknown signatures) VBS backdoor that dropped by exploit/shellcode. Additionally have used: Metasploit Meterpreter (in different configurations)  as a backdoor. Test cases simulated next kill chain steps: exploitation, downloading backdoor and execution.

Test case names:

TEST CASE	EXPECTED BEHAVIOR
EXPLOITATION/INTSALL: DbD with VBS - BoF	Blocked on any stage of attack (earlier better)
EXPLOITATION/INTSALL: DbD with VBS - UAF	Blocked on any stage of attack (earlier better)
EXPLOITATION/INTSALL: DbD with met64 - BoF	Blocked on any stage of attack (earlier better)
EXPLOITATION/INTSALL: DbD with met64 - UAF	Blocked on any stage of attack (earlier better)
INSTALL: VBS backdoor run	Blocked on any stage of attack (earlier better)
DELIVERY/INSTALL: Encoded Meterpereter x64 as an EXE drop	Detected and cleaned on delivery stage or blocked on execution
DELIVERY/INSTALL: Encoded Meterpereter x64 as a DLL drop	Detected and cleaned on delivery stage or blocked on execution
DELIVERY/INSTALL: Hash based malware	Detected and cleaned on delivery stage or blocked on execution
DELIVERY/INSTALL: Not Hash based malware	Detected and cleaned on delivery stage or blocked on execution

1.1.4       Command & Control

Custom VBS backdoor used stealth Reverse DNS tunnel (via nslookup) technique as a main communication method. Same method was used by DbD exploits for downloading backdoor (viaa svchost). Normal TCP connection was used as a communication method for Meterpreter. 

TEST CASE	EXPECTED BEHAVIOR
C&C: Meterpreter TCP	Detected or blocked
C&C: Reverse DNS via svchost	Detected

2.    Test files

Sorry, can't share samples, but could share just some sources from where I took some of them:

https://github.com/Cr4sh/fwexpl (ring3 -> ring0 -> SMM )
https://github.com/eik00d/Reverse_DNS_Shellcode (Reverse DNS shellcode, VBS PoC and C&C)
https://www.reverse.it (sometimes you could find good samples here)

3.    Results for 5 different products

TEST CASE\Solution	5 products
DELIVERY: HTML with exploits	0/5
DELIVERY: WORD with macros	0/5
EXPLOITTAION: ROP Shellcode	2/5 (StackPivot detection by hooks on VirtalAlloc/VirtualProtect), 1 of them bypassed, because hooks was on ring3, like it was https://asintsov.blogspot.de/2016/12/bypassing-exploit-protection-of-norton.html
EXPLOITTAION: Shellcode	5/5 (detection by hooks on various calls), 2 of them bypassed, because hooks was on ring3, like it was done there https://asintsov.blogspot.de/2016/12/bypassing-exploit-protection-of-norton.html
EXPLOITTAION: 1day ring0 exploit	0/5
EXPLOITTAION:  HeapSpray	1/5, by pre-allocation by well-known address: could be bypassed
EXPLOITTAION: Exploit triggers - UAF, Bof	0/5
EXPLOITATION/INTSALL: DbD with VBS - BoF	4/5  (EXPLOITTAION: ROP Shellcode + EXPLOITTAION: Shellcode + INSTALL: VBS backdoor run)
EXPLOITATION/INTSALL: DbD with VBS - UAF	5/5  (EXPLOITTAION: ROP Shellcode + EXPLOITTAION: Shellcode + INSTALL: VBS backdoor run)
EXPLOITATION/INTSALL: DbD with met64 - BoF	3/5  (EXPLOITTAION: ROP Shellcode + EXPLOITTAION: Shellcode + DELIVERY/INSTALL: Encoded Meterpereter x64 as a drop EXE)
EXPLOITATION/INTSALL: DbD with met64 - UAF	3/5  (EXPLOITTAION: ROP Shellcode + EXPLOITTAION: Shellcode + DELIVERY/INSTALL: Encoded Meterpereter x64 as a drop EXE)
EXPLOIT: 0day Macros run	2/5
EXPLOIT: Meterpreter  x86 (in mem)	1/5
INSTALL: VBS backdoor run	1/5
DELIVERY/INSTALL: Encoded Meterpereter x64 as a drop EXE	2/5 (AWESOME RESULTS)
DELIVERY/INSTALL: Encoded Meterpereter x64 as a drop DLL	3/5
DELIVERY/INSTALL: Hash based malware	4/5
DELIVERY/INSTALL: Not Hash based malware	2/5 (it is mean, if same malware will be changed and it will produce a new hash, than 2 products will miss it)
EXPLOIT: IOC bypass - migration to EXPLORER (CreateRemoteThread)	2/5 Blocked, and again hooks, 1 product has ring3 hooks and could be bypassed
C&C: Meterpreter TCP	3/5 will detect connection
C&C: Reverse DNS via svchost	0/5

Outcome

Of course my tests  does not reflect anything except the fact that those solution could pass or not  just those tests, exploits and backdoors chosen/created by me. On different set of exploits and samples or different attacks - we could get different results! But some general things are there:

• No one EPP solution could protect you from 100% attacks and threats! It is mean, if targeted attack (like APT) happen, and you are main target - EPP will not help you! 
• Some solutions do better in exploit prevention, but suck on malware detection, other more concentrated on "malware" detection, but can't stop exploits well
• Most exploit-protection/malware detection techniques used by EPP could be bypassed!
• Two different solution could stop same attack(exploit), and looks like result the same, but in fact they have stopped it on different stage, and that was what important for me: earlier - better 
• Vendors do not like public tests, that's why customers like me need to run those PoC/tests to choose right product by them-self. Do not trust "public" ratings, check solution that will fit your organisation and response your threats. Gartner magic quadrant or NSS reports does nor help much!
• Most EPP are cloud based... that creates additional problems for customers, especially if they collect to much data (like all system events on each endpoint or even files content)
• If you could use Win10 DeviceGuard  and control all binaries/scripts - it probably gives you better end-point security ;) 

UPD

Most typical NextGen "fails":

• "Corrupted" PE -> bypass checks/binary parsers
• EXE detected, but same product can't detect same malware in DLL...
• DLL detected, but same product can't detect same malware in EXE...
• Ring3 hooks (already mentioned, https://asintsov.blogspot.de/2016/12/bypassing-exploit-protection-of-norton.html )
• No signature checks... ok, but only VirusTotal hashes?
• IE->exec(CMD) - detected! IE->migrate(EXPLORER)->exec(CMD) - not

I checked, @matalz is right... delays also works! lol...

P.S. All vendors will fix at least things I reported, and they were very promising on new features. They not bad 8)