Applications demand fast and secure execution in diverse environments (Cloud data centers, Edge Nodes, mobile platforms etc.). Execution efficiency has been facilitated by the introduction of specialized compute elements (eg. GPUs), in order to accelerate specific parts of tasks/workloads (such is image processing). At the same time, too abstract deployment and management burdens, service providers use virtualization and container technologies. Eliminating the software overheads of these abstractions, especially in the context of hardware off-load/acceleration is a challenge and requires a number of factors to be taken into consideration: (a) portability, (b) performance, and (c) security. In this talk, we attack the first two factors and examine the option of unikernels and their surrounding ecosystem (application porting frameworks, orchestration frameworks, lightweight virtualization backends) in the context of hardware acceleration. We present our efforts in porting a novel hardware acceleration framework, vAccel, to the rumprun unikernel, digging into the internals of semantic abstraction for ML inference, as well as its implementation on rumprun and QEMU/KVM. We describe the frontend/backend driver port, the runtime needed to support the actual execution on the hardware and showcase our results in a brief demo of two unikernel frameworks performing ML inference on images.