1153123b90
Use a tmp file for swift writes
...
Else this leads to a seekable stream error with chunked uploads
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-03-04 13:11:59 +01:00
df55781814
Merge pull request #14254 from sparrowjack63/master
...
Dont check Bucket Name in Nextcloud
2019-03-01 13:46:04 +01:00
e3142a91b2
Bucket name - Alert in debug mode
...
Alert in debug mode only when trying to mount non compatible DNS bucket name (in order to not flood the logs)
Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
381ae44d38
Correct code indent
...
Just correct code indent.
Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
77473b0228
Warning on S3 not compatible DNS bucket name
...
The new created bucket should respect the DNS compatibility, nevertheless, Nextcloud should accept to mount "old created" buckets that does not respect DNS compatibility (Backward compatibility, or compatibility with CEPH).
Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
59c3d8d85f
Dont check Bucket Name in Nextcloud
...
Signed-off-by: Sebastien Cat <sparrow.jack63@gmail.com>
Signed-off-by: Sébastien Cat <sebastien.cat@inra.fr>
2019-03-01 11:28:54 +01:00
f6f002e2ad
Merge pull request #14210 from nextcloud/fix/14192/fix_empty_uploads
...
Fix empty file uploads to S3 (and other streaming storages)
2019-02-18 15:39:17 +01:00
104616c1da
Use CountWrapper instead of almost identical CountReadStream
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-15 13:24:58 +01:00
61d66d7954
Fix empty file uploads to S3
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2019-02-15 08:41:39 +01:00
d06c25b9c8
Update SwiftFactory.php
...
Auth-Service (in explizit v3) will recheck the cached-token and will end in an "token expired exception".
2019-02-13 12:26:27 +01:00
c26bdd4731
Fix: Check if `$this->params['user']` is an array
...
Signed-off-by: William Pain <pain.william@gmail.com>
2019-02-05 09:56:42 +01:00
1d322d3b37
always use multipart uploader for s3 uploads
...
the multipart uploader handles non seekable streams while `upload` does not
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-01-28 14:51:19 +01:00
5a27e54f4b
Merge pull request #13032 from nextcloud/objectstore-write-exists
...
upload new files in objectstore to a .part path first
2019-01-14 11:26:29 +01:00
615fb8cd77
Cache tokens when using swift's v2 authentication
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2019-01-13 19:59:48 +01:00
d6bf5d4384
upload new files in objectstore to a .part path first
...
This prevent the object store and cache from getting out of sync
when an objectstore silently fails or the php process get's killed
during the upload without giving us the chance to cleanup
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-12-12 15:24:40 +01:00
6c9f2644cf
Add objectExists to objectstore interface
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-12-12 14:17:19 +01:00
cef8eadf2a
Add PHPDoc for thrown exception
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2018-11-19 15:34:07 +01:00
35251928d5
forward object not found error in switch as dav 404
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-11-19 11:34:38 +01:00
93de63777e
extend storage api to allow directly writing a stream to storage
...
this removes the need for temporary storages with some external storage backends.
The new method is added to a separate interface to maintain compatibility with
storage backends implementing the storage interface directly (without inheriting common)
Currently the interface is implemented for objectstorage based storages and local storage
and used by webdav uploads
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-10-31 21:10:44 +01:00
47fd3499a3
Fix listing s3 objects when using google cloud aws compatibility
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-06-11 17:10:57 +02:00
ac26175a17
add azure unit tests with azurite
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-06-04 16:18:03 +02:00
60b7cbc0f0
add azure object storage backend
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2018-06-04 10:57:52 +02:00
a0e038f912
generateTokenFromCache is only valid for V3
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2018-04-26 20:51:05 +02:00
7318882357
json_decode an associative array for the swift cached token
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2018-04-26 20:47:47 +02:00
38a90130ce
move log constants to ILogger
...
Signed-off-by: Arthur Schiwon <blizzz@arthur-schiwon.de>
2018-04-26 10:45:52 +02:00
1ceb081c9b
Merge pull request #9280 from nextcloud/bugfix/noid/swift-v3-scope
...
Swift v3 requires setting a scope
2018-04-24 09:49:27 +02:00
ba71918ad2
Swift v3 requires setting a scope
...
Unscoped auth is not guaranteed to return a catalog. So require the
scope to be set properly.
See https://developer.openstack.org/api-ref/identity/v3/index.html#password-authentication-with-unscoped-authorization
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2018-04-23 19:34:09 +02:00
431ccb6386
Fix undefined variable - found by phan
...
Added in https://github.com/nextcloud/server/pull/8314/files#diff-a55c10804586cd797b7f3f16556c1457R157
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2018-04-23 11:34:41 +02:00
2f068bc9d1
Add more logging for the object storage during creation of the buckets
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2018-04-05 17:26:09 +02:00
63bc633d89
Merge pull request #8857 from nextcloud/swift-verify-cached-token
...
verify cached swift token
2018-03-19 10:10:04 +01:00
41954d2903
verify cached swift token
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-03-16 15:20:16 +01:00
d3e7996e80
Fix bucket and container usage
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2018-03-14 13:57:14 +01:00
4e8d5504d2
Properly cache swift v3 token
...
The V3 token has an export function. Without this the token is
effectively never cached.
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2018-03-12 20:53:50 +01:00
d05469959e
Merge pull request #8715 from nextcloud/objectstore-no-part-files
...
disable part files for object stores
2018-03-08 20:52:16 +01:00
fc35352122
disable part files for object stores
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-03-07 13:33:35 +01:00
9764e70494
don't read existing file when overwriting using object store
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-03-07 12:05:57 +01:00
812ea354d0
User proper username + pass
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2018-02-26 14:54:18 +01:00
31dd6d6aae
add support for v3 swift auth
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-02-26 14:54:08 +01:00
817f2f8f49
adjust swift external storage to new openstack sdk
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-02-19 19:41:16 +01:00
3192efc415
adjust swift object storage to new openstack sdk
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-02-19 19:41:16 +01:00
d3d045dd5c
Remove unused import statements
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2018-02-14 16:55:43 +01:00
3e0ef7e3f2
Try to silence phan
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2018-02-08 15:18:26 +01:00
e34c20af7d
don't swallow exception from touch
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-02-08 15:18:26 +01:00
85d23dd6fc
better error message on catalog error
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-02-08 15:18:26 +01:00
26e0c14c6d
Merge pull request #8033 from nextcloud/s3-uploader
...
Use S3Client::upload instead of splitting single/multipart upload ourselves
2018-02-08 14:30:02 +01:00
fd09c06eb4
actually return stream from swift
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-02-07 17:10:59 +01:00
a661f043e1
Remove unneeded semicolon and parentheses
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2018-01-26 23:46:40 +01:00
c005fc6755
Cleanup unused code
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2018-01-26 15:25:19 +01:00
3499cbdb76
Use S3Client::upload instead of splitting single/multipart upload ourselves
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2018-01-24 17:22:05 +01:00
2a38605545
Properly log the full exception instead of only the message
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2018-01-23 10:57:21 +01:00
34ced4dd97
add option to use legacy v2 auth with s3
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-12-19 11:54:55 +01:00
d2d73f1ce8
Also replace all other occurences
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2017-12-18 20:57:11 +01:00
046eb0c635
add retry wrapper when reading files from swift
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-12-04 16:34:53 +01:00
1ade6b0817
only create the file cache entry after the empty file was created successfully, otherwise file_exists() call on the initial file_put_content() will indicate that the file already exists
...
Signed-off-by: Bjoern Schiessle <bjoern@schiessle.org>
2017-11-17 12:45:46 +01:00
c566b2854b
Use non empty files for object store touch
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-11-08 17:51:02 +01:00
70a4860ccb
Fix contructing headers for s3 download
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-11-08 17:50:46 +01:00
31c5c2a592
Change @georgehrke's email
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2017-11-06 20:38:59 +01:00
da6c2c9da1
Merge pull request #7083 from nextcloud/update-license-headers
...
Update license headers
2017-11-06 18:48:04 +01:00
0eebff152a
Update license headers
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2017-11-06 16:56:19 +01:00
8b763ea0f7
Better error message for invalid bucket names
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2017-11-06 09:48:41 +01:00
3e0789aa94
Merge pull request #6958 from nextcloud/improved-mimetype-detection-for-object-storage
...
Improve mimetype detection for object storages
2017-11-01 11:04:15 +01:00
f9a7294807
use fopen directly when reading objects from s3
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-10-27 12:58:19 +02:00
405bbc1c61
Improve mimetype detection for object storages
...
Object storage instances always fall back to the content based mimetype detection, because the file name for object storage was always random due to the fact that it was temporarily storage in a generated temp file. This patch adds a check before that to make sure to use the original file name for this purpose and also remove possible other extensions like the versioning or part file extension.
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2017-10-26 16:31:59 +02:00
91b3536f45
propagate multipart upload exception when aborting upload
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-10-17 15:10:58 +02:00
e393b3553e
set s3 part size to 500mb
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-10-11 15:59:53 +02:00
385d6f098c
Add tests for multipart upload
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-09-21 14:47:34 +02:00
4ae46d8876
only do multipart upload for large files
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-09-21 14:06:59 +02:00
e4e5e735db
multipart upload for s3 object storage
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-09-20 17:51:58 +02:00
d526969a68
fix path style
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-09-18 15:16:28 +02:00
d70607104e
reuse object read/write/delete logic in s3 implementations
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-09-18 15:16:27 +02:00
dad18baec8
update aws sdk and move it to 3rdparty
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-09-18 15:16:27 +02:00
9a63ded43b
Fix uninitialized variable $this->params
...
Signed-off-by: William Pain <pain.william@gmail.com>
2017-09-11 10:01:12 +02:00
5f227bd93b
More phpstorm inspection fixes
...
Signed-off-by: Roeland Jago Douma <roeland@famdouma.nl>
2017-07-24 11:39:29 +02:00
0a12b5d38d
improved logging of objectore errors
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-06-21 17:33:26 -05:00
c54a59d51e
Remove unused use statements
...
Signed-off-by: Morris Jobke <hey@morrisjobke.de>
2017-04-22 19:23:31 -05:00
6991b79d40
serialize the token to json instead of using php's serialize
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-03-28 21:33:07 +02:00
205d5586e8
cache swift tokens in memcache
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-03-28 12:37:41 -06:00
8dbca71a77
better error messages for invalid regions, urltypes and service names
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-03-28 17:00:11 +02:00
5062d0ac50
better error messages when swift authentication fails
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-03-28 11:02:18 +02:00
5774d3e82c
replace close:// streamwrapper with CallBackWrapper
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2017-01-06 15:33:32 +01:00
3ab160dd5a
fopen s3 objects directly to work around unexplainable guzzle bug
...
For some reason when a text file started with a valid hex character ([0-9a-f]) it would eat the text untill the first newline
The new code does basically the same thing as guzzle/s3-sdk did only without wrapping everything in a guzzle stream
Signed-off-by: Robin Appelman <robin@icewind.nl>
2016-11-16 15:30:37 +01:00
eefd059716
add amazon s3 objectstore backend
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2016-11-16 15:30:36 +01:00
64e896cc0d
split testing of objectstoragestorage and objectstore implementations
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2016-11-16 15:30:36 +01:00
0a246f9852
split off s3 connection logic
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2016-11-16 15:30:36 +01:00
4a9361905d
make object prefix configurable
2016-11-14 15:10:56 +01:00
b72e5a2e4e
allow using 'bucket' to set the swift container
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2016-11-04 14:15:18 +01:00
049bcae29e
Allow setting a maximun number of buckets for multibucket
...
Signed-off-by: Robin Appelman <robin@icewind.nl>
2016-11-02 21:31:24 +01:00
0215b004da
Update with robin
2016-07-21 18:13:58 +02:00
ba87db3fcc
Fix others
2016-07-21 18:13:57 +02:00
aba539703c
Update license headers
2016-05-26 19:57:24 +02:00
5e2316d05d
Allow multibucket in objectstore
2016-05-23 20:42:08 +02:00
dedf392751
Move \OC\Files to PSR-4
2016-04-24 21:37:35 +02:00
Roeland Jago Douma
Roeland Jago Douma
S. Cat
Sebastien Cat
Volker
William Pain
Robin Appelman
Morris Jobke
Arthur Schiwon
Bjoern Schiessle
Joas Schilling
Jörn Friedrich Dreyer
Lukas Reschke
Roeland Jago Douma