forked from apache/cassandra-python-driver
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathquery_paging.html
More file actions
225 lines (187 loc) · 17.4 KB
/
Copy pathquery_paging.html
File metadata and controls
225 lines (187 loc) · 17.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta http-equiv="refresh" content="0; URL=https://docs.datastax.com/en/developer/python-driver/latest/query_paging/">
<link rel="canonical" href="https://docs.datastax.com/en/developer/python-driver/latest/query_paging/">
<title>Paging Large Queries — Cassandra Driver 3.13.0 documentation</title>
<link rel="stylesheet" href="_static/custom.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: './',
VERSION: '3.13.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Lightweight Transactions (Compare-and-set)" href="lwt.html" />
<link rel="prev" title="Performance Notes" href="performance.html" />
<link rel="stylesheet" href="_static/custom.css" type="text/css" />
<meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
</head>
<body>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="paging-large-queries">
<span id="query-paging"></span><h1>Paging Large Queries<a class="headerlink" href="#paging-large-queries" title="Permalink to this headline">¶</a></h1>
<p>Cassandra 2.0+ offers support for automatic query paging. Starting with
version 2.0 of the driver, if <a class="reference internal" href="api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version" title="cassandra.cluster.Cluster.protocol_version"><code class="xref py py-attr docutils literal"><span class="pre">protocol_version</span></code></a> is greater than
<code class="xref py py-const docutils literal"><span class="pre">2</span></code> (it is by default), queries returning large result sets will be
automatically paged.</p>
<div class="section" id="controlling-the-page-size">
<h2>Controlling the Page Size<a class="headerlink" href="#controlling-the-page-size" title="Permalink to this headline">¶</a></h2>
<p>By default, <a class="reference internal" href="api/cassandra/cluster.html#cassandra.cluster.Session.default_fetch_size" title="cassandra.cluster.Session.default_fetch_size"><code class="xref py py-attr docutils literal"><span class="pre">Session.default_fetch_size</span></code></a> controls how many rows will
be fetched per page. This can be overridden per-query by setting
<a class="reference internal" href="api/cassandra/query.html#cassandra.query.Statement.fetch_size" title="cassandra.query.Statement.fetch_size"><code class="xref py py-attr docutils literal"><span class="pre">fetch_size</span></code></a> on a <a class="reference internal" href="api/cassandra/query.html#cassandra.query.Statement" title="cassandra.query.Statement"><code class="xref py py-class docutils literal"><span class="pre">Statement</span></code></a>. By default, each page
will contain at most 5000 rows.</p>
</div>
<div class="section" id="handling-paged-results">
<h2>Handling Paged Results<a class="headerlink" href="#handling-paged-results" title="Permalink to this headline">¶</a></h2>
<p>Whenever the number of result rows for are query exceed the page size, an
instance of <code class="xref py py-class docutils literal"><span class="pre">PagedResult</span></code> will be returned instead of a normal
list. This class implements the iterator interface, so you can treat
it like a normal iterator over rows:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cassandra.query</span> <span class="k">import</span> <span class="n">SimpleStatement</span>
<span class="n">query</span> <span class="o">=</span> <span class="s2">"SELECT * FROM users"</span> <span class="c1"># users contains 100 rows</span>
<span class="n">statement</span> <span class="o">=</span> <span class="n">SimpleStatement</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">fetch_size</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
<span class="k">for</span> <span class="n">user_row</span> <span class="ow">in</span> <span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">statement</span><span class="p">):</span>
<span class="n">process_user</span><span class="p">(</span><span class="n">user_row</span><span class="p">)</span>
</pre></div>
</div>
<p>Whenever there are no more rows in the current page, the next page will
be fetched transparently. However, note that it <em>is</em> possible for
an <code class="xref py py-class docutils literal"><span class="pre">Exception</span></code> to be raised while fetching the next page, just
like you might see on a normal call to <code class="docutils literal"><span class="pre">session.execute()</span></code>.</p>
<p>If you use <a class="reference internal" href="api/cassandra/cluster.html#cassandra.cluster.Session.execute_async" title="cassandra.cluster.Session.execute_async"><code class="xref py py-meth docutils literal"><span class="pre">Session.execute_async()</span></code></a> along with,
<a class="reference internal" href="api/cassandra/cluster.html#cassandra.cluster.ResponseFuture.result" title="cassandra.cluster.ResponseFuture.result"><code class="xref py py-meth docutils literal"><span class="pre">ResponseFuture.result()</span></code></a>, the first page will be fetched before
<a class="reference internal" href="api/cassandra/cluster.html#cassandra.cluster.ResponseFuture.result" title="cassandra.cluster.ResponseFuture.result"><code class="xref py py-meth docutils literal"><span class="pre">result()</span></code></a> returns, but latter pages will be
transparently fetched synchronously while iterating the result.</p>
</div>
<div class="section" id="handling-paged-results-with-callbacks">
<h2>Handling Paged Results with Callbacks<a class="headerlink" href="#handling-paged-results-with-callbacks" title="Permalink to this headline">¶</a></h2>
<p>If callbacks are attached to a query that returns a paged result,
the callback will be called once per page with a normal list of rows.</p>
<p>Use <a class="reference internal" href="api/cassandra/cluster.html#cassandra.cluster.ResponseFuture.has_more_pages" title="cassandra.cluster.ResponseFuture.has_more_pages"><code class="xref py py-attr docutils literal"><span class="pre">ResponseFuture.has_more_pages</span></code></a> and
<a class="reference internal" href="api/cassandra/cluster.html#cassandra.cluster.ResponseFuture.start_fetching_next_page" title="cassandra.cluster.ResponseFuture.start_fetching_next_page"><code class="xref py py-meth docutils literal"><span class="pre">ResponseFuture.start_fetching_next_page()</span></code></a> to continue fetching
pages. For example:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">PagedResultHandler</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">future</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">error</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">finished_event</span> <span class="o">=</span> <span class="n">Event</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">future</span> <span class="o">=</span> <span class="n">future</span>
<span class="bp">self</span><span class="o">.</span><span class="n">future</span><span class="o">.</span><span class="n">add_callbacks</span><span class="p">(</span>
<span class="n">callback</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">handle_page</span><span class="p">,</span>
<span class="n">errback</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">handle_err</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">handle_page</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">rows</span><span class="p">):</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
<span class="n">process_row</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">future</span><span class="o">.</span><span class="n">has_more_pages</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">future</span><span class="o">.</span><span class="n">start_fetching_next_page</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">finished_event</span><span class="o">.</span><span class="n">set</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">handle_error</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exc</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">error</span> <span class="o">=</span> <span class="n">exc</span>
<span class="bp">self</span><span class="o">.</span><span class="n">finished_event</span><span class="o">.</span><span class="n">set</span><span class="p">()</span>
<span class="n">future</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">execute_async</span><span class="p">(</span><span class="s2">"SELECT * FROM users"</span><span class="p">)</span>
<span class="n">handler</span> <span class="o">=</span> <span class="n">PagedResultHandler</span><span class="p">(</span><span class="n">future</span><span class="p">)</span>
<span class="n">handler</span><span class="o">.</span><span class="n">finished_event</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">if</span> <span class="n">handler</span><span class="o">.</span><span class="n">error</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">handler</span><span class="o">.</span><span class="n">error</span>
</pre></div>
</div>
</div>
<div class="section" id="resume-paged-results">
<h2>Resume Paged Results<a class="headerlink" href="#resume-paged-results" title="Permalink to this headline">¶</a></h2>
<p>You can resume the pagination when executing a new query by using the <a class="reference internal" href="api/cassandra/cluster.html#cassandra.cluster.ResultSet.paging_state" title="cassandra.cluster.ResultSet.paging_state"><code class="xref py py-attr docutils literal"><span class="pre">ResultSet.paging_state</span></code></a>. This can be useful if you want to provide some stateless pagination capabilities to your application (ie. via http). For example:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cassandra.query</span> <span class="k">import</span> <span class="n">SimpleStatement</span>
<span class="n">query</span> <span class="o">=</span> <span class="s2">"SELECT * FROM users"</span>
<span class="n">statement</span> <span class="o">=</span> <span class="n">SimpleStatement</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">fetch_size</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">statement</span><span class="p">)</span>
<span class="c1"># save the paging_state somewhere and return current results</span>
<span class="n">session</span><span class="p">[</span><span class="s1">'paging_stage'</span><span class="p">]</span> <span class="o">=</span> <span class="n">results</span><span class="o">.</span><span class="n">paging_state</span>
<span class="c1"># resume the pagination sometime later...</span>
<span class="n">statement</span> <span class="o">=</span> <span class="n">SimpleStatement</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">fetch_size</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span>
<span class="n">ps</span> <span class="o">=</span> <span class="n">session</span><span class="p">[</span><span class="s1">'paging_state'</span><span class="p">]</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">statement</span><span class="p">,</span> <span class="n">paging_state</span><span class="o">=</span><span class="n">ps</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h1 class="logo"><a href="index.html">Cassandra Driver</a></h1>
<p class="blurb">Python driver for Cassandra</p>
<p>
<iframe src="https://ghbtns.com/github-btn.html?user=datastax&repo=python-driver&type=star&count=true&size=large&v=2"
allowtransparency="true" frameborder="0" scrolling="0" width="200px" height="35px"></iframe>
</p>
<h3>Navigation</h3>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="api/index.html">API Documentation</a></li>
<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="getting_started.html">Getting Started</a></li>
<li class="toctree-l1"><a class="reference internal" href="upgrading.html">Upgrading</a></li>
<li class="toctree-l1"><a class="reference internal" href="execution_profiles.html">Execution Profiles</a></li>
<li class="toctree-l1"><a class="reference internal" href="performance.html">Performance Notes</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Paging Large Queries</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#controlling-the-page-size">Controlling the Page Size</a></li>
<li class="toctree-l2"><a class="reference internal" href="#handling-paged-results">Handling Paged Results</a></li>
<li class="toctree-l2"><a class="reference internal" href="#handling-paged-results-with-callbacks">Handling Paged Results with Callbacks</a></li>
<li class="toctree-l2"><a class="reference internal" href="#resume-paged-results">Resume Paged Results</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="lwt.html">Lightweight Transactions (Compare-and-set)</a></li>
<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="user_defined_types.html">User Defined Types</a></li>
<li class="toctree-l1"><a class="reference internal" href="object_mapper.html">Object Mapper</a></li>
<li class="toctree-l1"><a class="reference internal" href="dates_and_times.html">Working with Dates and Times</a></li>
<li class="toctree-l1"><a class="reference internal" href="faq.html">Frequently Asked Questions</a></li>
</ul>
<div class="relations">
<h3>Related Topics</h3>
<ul>
<li><a href="index.html">Documentation overview</a><ul>
<li>Previous: <a href="performance.html" title="previous chapter">Performance Notes</a></li>
<li>Next: <a href="lwt.html" title="next chapter">Lightweight Transactions (Compare-and-set)</a></li>
</ul></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<div><input type="text" name="q" /></div>
<div><input type="submit" value="Go" /></div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
©2013-2017 DataStax.
|
Powered by <a href="http://sphinx-doc.org/">Sphinx 1.6.6</a>
& <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a>
|
<a href="_sources/query_paging.rst.txt"
rel="nofollow">Page source</a>
</div>
</body>
</html>