1/* Part of SWI-Prolog 2 3 Author: Jan Wielemaker 4 E-mail: J.Wielemaker@vu.nl 5 WWW: http://www.swi-prolog.org 6 Copyright (c) 2002-2016, University of Amsterdam, 7 VU University Amsterdam 8 All rights reserved. 9 10 Redistribution and use in source and binary forms, with or without 11 modification, are permitted provided that the following conditions 12 are met: 13 14 1. Redistributions of source code must retain the above copyright 15 notice, this list of conditions and the following disclaimer. 16 17 2. Redistributions in binary form must reproduce the above copyright 18 notice, this list of conditions and the following disclaimer in 19 the documentation and/or other materials provided with the 20 distribution. 21 22 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 25 FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 26 COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 27 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 28 BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 30 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 31 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 32 ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 33 POSSIBILITY OF SUCH DAMAGE. 34*/ 35 36:- module(http_client, 37 [ http_get/3, % +URL, -Reply, +Options 38 http_delete/3, % +URL, -Reply, +Options 39 http_post/4, % +URL, +In, -Reply, +Options 40 http_put/4, % +URL, +In, -Reply, +Options 41 http_patch/4, % +URL, +In, -Reply, +Options 42 http_read_data/3, % +Header, -Data, :Options 43 http_disconnect/1 % +What 44 ]). 45:- autoload(http_header,[http_post_data/3]). 46:- autoload(http_stream,[http_chunked_open/3,stream_range_open/3]). 47:- autoload(library(error),[must_be/2]). 48:- autoload(library(lists),[delete/3,select/3]). 49:- autoload(library(memfile), 50 [ new_memory_file/1, open_memory_file/4, free_memory_file/1, 51 memory_file_to_atom/3, memory_file_to_string/3, 52 memory_file_to_codes/3, open_memory_file/3 53 ]). 54:- autoload(library(option), 55 [option/3,option/2,meta_options/3,select_option/3]). 56:- autoload(library(uri),[uri_query_components/2]). 57:- autoload(library(http/http_open), 58 [http_open/3,http_close_keep_alive/1]). 59 60:- meta_predicate 61 http_read_data(, , ). 62 63:- multifile 64 http_convert_data/4, % http_read_data plugin-hook 65 http:post_data_hook/3. 66 67:- predicate_options(http_get/3, 3, 68 [ pass_to(http_open/3, 3), 69 pass_to(http_read_data/3, 3) 70 ]). 71:- predicate_options(http_delete/3, 3, [pass_to(http_get/3, 3)]). 72:- predicate_options(http_post/4, 4, [pass_to(http_get/3, 3)]). 73:- predicate_options(http_put/4, 4, [pass_to(http_post/4, 4)]). 74:- predicate_options(http_read_data/3, 3, 75 [ to(any), 76 content_type(any), 77 form_data(oneof([form,mime])), 78 input_encoding(encoding), 79 on_filename(callable) 80 ]). 81 82 83/** <module> HTTP client library 84 85This library provides the four basic HTTP client actions: =GET=, 86=DELETE=, =POST= and =PUT=. In addition, it provides http_read_data/3, 87which is used by library(http/http_parameters) to decode =POST= data in 88server applications. 89 90This library is based on http_open/3, which opens a URL as a Prolog 91stream. The reply is processed by http_read_data/3. The following 92content-types are supported. Options passed to http_get/3 and friends 93are passed to http_read_data/3, which in turn passes them to the 94conversion predicates. Support for additional content types can be added 95by extending the multifile predicate http_client:http_convert_data/4. 96 97 - 'application/x-www-form-urlencoded' 98 Built in. Converts form-data into a list of `Name=Value` terms. 99 - 'application/x-prolog' 100 Built in. Reads a single Prolog term. 101 - 'multipart/form-data' 102 Processed if library(http/http_multipart_plugin) is loaded. This 103 format should be used to handle web forms that upload a file. 104 - 'text/html' | 'text/xml' 105 Processed if library(http/http_sgml_plugin) is loaded. See load_html/3 106 for details and load_xml/3 for details. The output is often processed 107 using xpath/3. 108 - 'application/json' | 'application/jsonrequest' 109 Processed if library(http/http_json) is loaded. The option 110 json_object(As) can be used to return a term json(Attributes) 111 (`As` is `term`) or a dict (`As` is `dict`). 112*/ 113 114 /******************************* 115 * GET * 116 *******************************/ 117 118%! http_get(+URL, -Data, +Options) is det. 119% 120% Get data from a URL server and convert it to a suitable Prolog 121% representation based on the =|Content-Type|= header and plugins. 122% This predicate is the common implementation of the HTTP client 123% operations. The predicates http_delete/3, http_post/4 and 124% http_put/4 call this predicate with an appropriate 125% method(+Method) option and ---for http_post/4 and http_put/4--- 126% a post(+Data) option. 127% 128% Options are passed to http_open/3 and http_read_data/3. Other 129% options: 130% 131% - reply_header(-Fields) 132% Synonym for headers(Fields) from http_open/3. Provided for 133% backward compatibility. Note that http_version(Major-Minor) 134% is missing in the new version. 135 136http_get(URL, Data, Options) :- 137 headers_option(Options, Options1, Headers), 138 option(reply_header(Headers), Options, _), 139 http_open(URL, In, Options1), 140 delete(Headers, transfer_encoding(_), Headers1), 141 call_cleanup( 142 http_read_data(In, Headers1, Data, Options), 143 close(In)). 144 145headers_option(Options, Options1, Headers) :- 146 option(headers(Headers), Options), 147 !, 148 Options1 = Options. 149headers_option(Options, [headers(Headers)|Options], Headers). 150 151 152%! http_delete(+URL, -Data, +Options) is det. 153% 154% Execute a =DELETE= method on the server. Arguments are the same 155% as for http_get/3. Typically one should pass the option 156% status_code(-Code) to assess and evaluate the returned status 157% code. Without, codes other than 200 are interpreted as an error. 158% 159% @tbd Properly map the 201, 202 and 204 replies. 160% @see Implemented on top of http_get/3. 161 162http_delete(URL, Data, Options) :- 163 http_get(URL, Data, [method(delete)|Options]). 164 165 166%! http_post(+URL, +Data, -Reply, +Options) is det. 167% 168% Issue an HTTP =POST= request. Data is posted using 169% http_post_data/3. The HTTP server reply is returned in Reply, 170% using the same rules as for http_get/3. 171% 172% @see Implemented on top of http_get/3. 173 174http_post(URL, Data, Reply, Options) :- 175 http_get(URL, Reply, 176 [ post(Data) 177 | Options 178 ]). 179 180%! http_put(+URL, +Data, -Reply, +Options) 181% 182% Issue an HTTP =PUT= request. Arguments are the same as for 183% http_post/4. 184% 185% @see Implemented on top of http_post/4. 186 187http_put(URL, In, Out, Options) :- 188 http_post(URL, In, Out, [method(put)|Options]). 189 190%! http_patch(+URL, +Data, -Reply, +Options) 191% 192% Issue an HTTP =PATCH= request. Arguments are the same as for 193% http_post/4. 194% 195% @see Implemented on top of http_post/4. 196 197http_patch(URL, In, Out, Options) :- 198 http_post(URL, In, Out, [method(patch)|Options]). 199 200%! http_read_data(+Request, -Data, +Options) is det. 201% 202% Read data from an HTTP connection and convert it according to 203% the supplied to(Format) option or based on the =|Content-type|= 204% in the Request. The following options are supported: 205% 206% * to(Format) 207% Convert data into Format. Values are: 208% - stream(+WriteStream)) 209% Append the content of the message to Stream 210% - atom 211% Return the reply as an atom 212% - string 213% Return the reply as a string 214% - codes 215% Return the reply as a list of codes 216% * form_data(AsForm) 217% * input_encoding(+Encoding) 218% * on_filename(:CallBack) 219% These options are implemented by the plugin 220% library(http/http_multipart_plugin) and apply to processing 221% =|multipart/form-data|= content. 222% * content_type(+Type) 223% Overrule the content-type that is part of Request as a 224% work-around for wrongly configured servers. 225% 226% Without plugins, this predicate handles 227% 228% * 'application/x-www-form-urlencoded' 229% Converts form-data into a list of `Name=Value` terms. 230% * 'application/x-prolog' 231% Converts data into a Prolog term. 232% 233% @param Request is a parsed HTTP request as returned by 234% http_read_request/2 or available from the HTTP server's request 235% dispatcher. Request must contain a term input(In) that provides 236% the input stream from the HTTP server. 237 238http_read_data(Fields, Data, QOptions) :- 239 meta_options(is_meta, QOptions, Options), 240 memberchk(input(In), Fields), 241 ( http_read_data(In, Fields, Data, Options) 242 -> true 243 ; throw(error(failed(http_read_data), _)) 244 ). 245 246is_meta(on_filename). 247 248http_read_data(In, Fields, Data, Options) :- % Transfer-encoding: chunked 249 select(transfer_encoding(chunked), Fields, RestFields), 250 !, 251 setup_call_cleanup( 252 http_chunked_open(In, DataStream, []), 253 http_read_data(DataStream, RestFields, Data, Options), 254 close(DataStream)). 255http_read_data(In, Fields, Data, Options) :- 256 option(to(X), Options), 257 !, 258 ( X = stream(Stream) 259 -> ( memberchk(content_length(Bytes), Fields) 260 -> copy_stream_data(In, Stream, Bytes) 261 ; copy_stream_data(In, Stream) 262 ) 263 ; must_be(oneof([atom,string,codes]), X), 264 setup_call_cleanup( 265 new_memory_file(MemFile), 266 ( setup_call_cleanup( 267 open_memory_file(MemFile, write, Stream, 268 [encoding(octet)]), 269 ( memberchk(content_length(Bytes), Fields) 270 -> copy_stream_data(In, Stream, Bytes) 271 ; copy_stream_data(In, Stream) 272 ), 273 close(Stream)), 274 encoding(Fields, Encoding, Options), 275 memory_file_to(X, MemFile, Encoding, Data0) 276 ), 277 free_memory_file(MemFile)), 278 Data = Data0 279 ). 280http_read_data(In, Fields, Data, _) :- 281 option(content_type(ContentType), Fields), 282 is_content_type(ContentType, 'application/x-www-form-urlencoded'), 283 !, 284 http_read_data(In, Fields, Codes, [to(string)]), 285 uri_query_components(Codes, Data). 286http_read_data(In, Fields, Data, Options) :- % call hook 287 ( select_option(content_type(Type), Options, Options1) 288 -> delete(Fields, content_type(_), Fields1), 289 http_convert_data(In, [content_type(Type)|Fields1], Data, Options1) 290 ; http_convert_data(In, Fields, Data, Options) 291 ), 292 !. 293http_read_data(In, Fields, Data, Options) :- 294 http_read_data(In, Fields, Data, [to(atom)|Options]). 295 296memory_file_to(atom, MemFile, Encoding, Data) :- 297 memory_file_to_atom(MemFile, Data, Encoding). 298memory_file_to(string, MemFile, Encoding, Data) :- 299 memory_file_to_string(MemFile, Data, Encoding). 300memory_file_to(codes, MemFile, Encoding, Data) :- 301 memory_file_to_codes(MemFile, Data, Encoding). 302 303 304encoding(_Fields, Encoding, Options) :- 305 option(input_encoding(Encoding), Options), 306 !. 307encoding(Fields, utf8, _) :- 308 memberchk(content_type(Type), Fields), 309 ( sub_atom(Type, _, _, _, 'UTF-8') 310 -> true 311 ; sub_atom(Type, _, _, _, 'utf-8') 312 ), 313 !. 314encoding(_, octet, _). 315 316is_content_type(ContentType, Check) :- 317 sub_atom(ContentType, 0, Len, After, Check), 318 ( After == 0 319 -> true 320 ; sub_atom(ContentType, Len, 1, _, ';') 321 ). 322 323%! http_convert_data(+In, +Fields, -Data, +Options) is semidet. 324% 325% Multi-file hook to convert a HTTP payload according to the 326% _Content-Type_ header. The default implementation deals with 327% application/x-prolog. The HTTP framework provides 328% implementations for JSON (library(http/http_json)), HTML/XML 329% (library(http/http_sgml_plugin)) 330 331http_convert_data(In, Fields, Data, Options) :- 332 memberchk(content_type(Type), Fields), 333 is_content_type(Type, 'application/x-prolog'), 334 !, 335 ( memberchk(content_length(Bytes), Fields) 336 -> setup_call_cleanup( 337 ( stream_range_open(In, Range, [size(Bytes)]), 338 set_stream(Range, encoding(utf8)), 339 set_stream(Range, file_name('HTTP:DATA')) 340 ), 341 read_term(Range, Data, Options), 342 close(Range)) 343 ; set_stream(In, encoding(utf8)), 344 read_term(In, Data, Options) 345 ). 346 347%! http_disconnect(+Connections) is det. 348% 349% Close down some connections. Currently Connections must have the 350% value =all=, closing all connections. 351% 352% @deprecated New code should use http_close_keep_alive/1 from 353% library(http/http_open). 354 355http_disconnect(all) :- 356 http_close_keep_alive(_). 357 358%! http:post_data_hook(+Term, +Out, +Options) is semidet. 359% 360% Hook to extend the datatypes supported by the post(Data) option 361% of http_open/3. The default implementation supports 362% prolog(Term), sending a Prolog term as =|application/x-prolog|=. 363 364httppost_data_hook(prolog(Term), Out, HdrExtra) :- 365 setup_call_cleanup( 366 ( new_memory_file(MemFile), 367 open_memory_file(MemFile, write, Handle) 368 ), 369 ( format(Handle, 370 'Content-Type: application/x-prolog; charset=UTF-8~n~n', 371 []), 372 write_term(Handle, Term, 373 [ quoted(true), 374 ignore_ops(true), 375 fullstop(true), 376 nl(true) 377 ]) 378 ), 379 close(Handle)), 380 setup_call_cleanup( 381 open_memory_file(MemFile, read, RdHandle, 382 [ free_on_close(true) 383 ]), 384 http_post_data(cgi_stream(RdHandle), Out, HdrExtra), 385 close(RdHandle))