In the Marketo landng page i have created a button "Download case study" when we click on the button it should display the form pop up its working fine till here. Below is the code for reference.
Reference code :
<a data-toggle="modal" data-target="#myModal" id="formButton" mktoName="Form Button">Download case study</a></div>
<div class="modal fade" id="myModal" tabindex="-1" role="dialog" aria-labelledby="myModalLabel">
<div class="modal-dialog" role="document">
<button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">×</span></button>
<h4 class="modal-title" id="myModalLabel">Complete This Form</h4>
<div class="col-md-12 text-center">
<div class='mktoForm' id='theForm' mktoName='Asset Form' style="padding-top: 20px; padding-bottom: 20px;">
Forget about jQuery.
And some examples will definitely help you: http://developers.marketo.com/rest-api/assets/forms/examples/ and especially the one titled "Take Visitor to User-Defined URL"
In addition, when submitting such questions always:
 Provide the URL of your runnable code (the HTML you've provided is of no use because it doesn't even contain the renderedtag.
 If posting code inline, highlight it using the Advanced Editor's syntax highlighter:
Can't you just tell the Form to redirect to the PDF instead? Not sure you need special code unless you want a new window.
Yes and no!
Naveen is implying that he's using the MktoForms2 lightbox feature (at least I think so, it hasn't been shown yet). Because that has its own subset of the Forms API event model I want to see that it's set up correctly.
The wider problem is that it's actually impossible to force a download of a PDF with Marketo forms, as you know. So unfortunately the precise answer is "You can't do this," although you can have a PDF either download or open.