I have had two bikes that did that from box stock to every lifter on the planet including our own.. SOme times it a case where the oil will be able to bleed / leak down. the spring pressure bleeds the lifter down you fire it and their is air in the system and you induce that air in the the lifter then it takes a bit to fully pump up .
My 12 SG was that way new.. HD installed under warranty two sets of lifters one stock one SE new cam plate and pump , new rockers , supports and shafts. It would always do it .. Today HD stands by the fact that a hyd lifter can bleed down and its not a issue.. In my case I did find the issue with that engine when I pulled it.. I made o ring plugs for the lifter bores and put air pressure to the case.. cold it was fine but when I heated the case up it had a porosity issue and it would leak internal .. This is not a common issues but with castings it does happen.. Had a transmission in my chevy truck have this issue and they had to replace the case ( well they gave me a new unit) but to the point it can be other things.. Lifter is a simple item to try and replace if it solves it then great if not and you try a few sets of other brands it may be something else